Data Engineer - Streaming (WkStream 2 - Kafka)
Tasks
- Apply PII detection and Protegrity tokenization hooks
- Build PySpark Structured Streaming applications
- Configure Kafka source parameters
- Design streaming ingestion pipelines
- Handle errors with dead letter queues
- Implement foreachBatch micro batch writes
- Ingest data from Kafka topics
- Map schemas and write to Apache Iceberg
- Parse JSON and Avro payloads
- Recover from checkpoint failures
- Set up checkpoint and offset management
- Support UAT and review compliance
- Validate schema and row counts
- Verify Kafka offset commits
- Write unit and integration tests
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Avro | Apache Iceberg | Apache Kafka | Apache Spark | Checkpoints | Confluent Kafka | Dead Letter Queue | ForeachBatch | JSON | Java | Kafka Connect | PySpark | S3 | Scala | Structured Streaming | Watermarking
Education
N/A
Roles
Related jobs
-
Associate Process Manager INR 1800K-2800KAgentic AI | Agile methodologies | Causal Inference | Cypher | Experimental DesignMid-level Full TimeMumbai, Maharashtra, India3h ago
-
Partner Engineer, Generative AI INR 2500K-3000KAI infrastructure | AWS | Azure | C plus plus | Cloud platformSenior-level Full TimeBangalore, India | Mumbai, India3h ago
-
Principal Consultant - Data & AI INR 2500K-3200KAI Services | Agile | Apache Kafka | Apache Spark | Artificial IntelligenceSenior-level Full TimeHyderabad, TS, IN; Bengaluru, KA, IN; …3h ago
-
IT Support Specialist 1 (Regional) INR 2000K-2000KAgile | Apache Spark | Azure Data | Azure Data Factory | Azure DevOpsSenior-level Full TimeBengaluru, KA, IN6h ago
-
Sr. Systems Engineer (3 - 6 years, Kafka, IBM MQ, DevOps, Cloud, Gen AI, Automation) INR 2535K-4000KApache Kafka | KSQLDB | Kafka | Kafka Connect | Kafka StreamsHybrid workSenior-level Full TimeBengaluru, INDIA, India9h ago
-
Software Engineer (6months to 2 Years of experience - JAVA/Python Backend, CICD, APIs, GEN AI, Microservices, Cloud) INR 1430K-2000KAgile | Apache Kafka | Automated testing | C++ | CI/CDContinued learning opportunities | MentorshipEntry-level Full TimeBengaluru, INDIA, India10h ago
-
Data Engineer INR 1500K-1800KAWS | Amazon Kinesis | Apache Airflow | Apache Beam | Apache FlinkGlobal induction program | Talent development programs | Time off for digital disconnect days | Time off for volunteering | Wellbeing programsMid-level Full TimeBengaluru, KA, India12h ago
-
Apache Airflow | Apache Hadoop | Apache Spark | App Service | Azure AppFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeKolkata DN 57, India14h ago
-
Apache Airflow | Apache Spark | Azure Cosmos | Azure Cosmos DB | Azure DataSenior-level Full TimeKolkata DN 57, India14h ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache Spark | App ServiceSenior-level Full TimeKolkata DN 57, India14h ago
-
API Integration | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India14h ago
-
Amazon Web Services | Apache Airflow | Apache Databricks | Apache Hadoop | Apache SparkSenior-level Full TimeKolkata DN 57, India14h ago
-
API ingestion | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India14h ago
-
Ab Initio | Apache Airflow | BigQuery | Cloud Storage | Cloud platformFlexible work arrangements | Inclusive workplace culture | MentorshipSenior-level Full TimeBengaluru Millenia, India14h ago
-
Ab Initio | BigQuery | Cloud Composer | Cloud Storage | Cloud platformFlexible work programs | Inclusive workplace culture | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India14h ago
-
Ab Initio | Apache Airflow | BigQuery | Cloud Storage | Cloud platformFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India14h ago
-
Ab Initio | Apache Dataflow | BigQuery | Cloud Composer | Cloud StorageCompetitive compensation | Flexibility programmes | Hybrid work environment | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India14h ago
-
AWS Data | AWS Data Pipeline | AWS Glue | AWS Lake Formation | AWS LambdaFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India14h ago
-
AWS Data | AWS Data Pipeline | AWS Glue | AWS IAM | AWS KMSSenior-level Full TimeBengaluru Millenia, India14h ago
-
AWS Data | AWS Data Pipeline | AWS Glue | AWS IAM | AWS Lake FormationFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India14h ago
-
Ab Initio | Apache Airflow | BigQuery | Cloud Storage | Cloud platformFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India14h ago
-
Ab Initio | BigQuery | Cloud Storage | Cloud platform | ComposerFlexibility programmes | Global exposure | Hybrid work environment | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India14h ago
-
Apache Airflow | Apache Hadoop | Apache Spark | App Service | Azure AppFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeKolkata DN 57, India14h ago
-
API ingestion | Apache Airflow | Apache Kafka | Apache Spark | App ServiceFlexible work programs | Inclusive benefits | Mentorship | Work-life balanceSenior-level Full TimeKolkata DN 57, India14h ago
-
API ingestion | Agile | Apache Airflow | Apache Hadoop | Apache SparkFlexibility programs | Inclusive benefits | MentorshipSenior-level Full TimeKolkata DN 57, India14h ago