Data Engineer - Streaming (WkStream 2 - Kafka)
Tasks
- Apply PII detection and Protegrity tokenization hooks
- Build PySpark Structured Streaming applications
- Configure Kafka source parameters
- Design streaming ingestion pipelines
- Handle errors with dead letter queues
- Implement foreachBatch micro batch writes
- Ingest data from Kafka topics
- Map schemas and write to Apache Iceberg
- Parse JSON and Avro payloads
- Recover from checkpoint failures
- Set up checkpoint and offset management
- Support UAT and review compliance
- Validate schema and row counts
- Verify Kafka offset commits
- Write unit and integration tests
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Avro | Apache Iceberg | Apache Kafka | Apache Spark | Checkpoints | Confluent Kafka | Dead Letter Queue | ForeachBatch | JSON | Java | Kafka Connect | PySpark | S3 | Scala | Structured Streaming | Watermarking
Education
N/A
Roles
Related jobs
-
Assistant Manager Machine Learning INR 1500K-1809KA/B | A/B Testing | AWS Glue | AWS Lambda | AWS SageMakerMid-level Full TimeIndia-Gurugram3h ago
-
Senior-level Full TimeKochi, India3h ago
-
Senior-level Full TimeIN-TN-Chennai3h ago
-
Associate Process Manager INR 1800K-2800KAgentic AI | Agile methodologies | Causal Inference | Cypher | Experimental DesignMid-level Full TimeMumbai, Maharashtra, India4h ago
-
Partner Engineer, Generative AI INR 2500K-3000KAI infrastructure | AWS | Azure | C plus plus | Cloud platformSenior-level Full TimeBangalore, India | Mumbai, India4h ago
-
Entry-level Full TimePune, India4h ago
-
Principal Consultant - Data & AI INR 2500K-3200KAI Services | Agile | Apache Kafka | Apache Spark | Artificial IntelligenceSenior-level Full TimeHyderabad, TS, IN; Bengaluru, KA, IN; …5h ago
-
Senior Data Engineer INR 2500K-2829KApache Airflow | Apache Spark | Azure | Azure Data | Azure Data LakeEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offSenior-level Full TimeChennai, TN, India5h ago
-
Senior Associate -Applied AI ML -Digital INR 1050K-1496KAlgorithms | Apache Spark | Big Data | Classification | CompilersMid-level Full TimeBengaluru, Karnataka, India6h ago
-
Sr. SW Engineer - SDET - Automation, GenAI tooling, API Testing INR 2156K-3000KAI | AI Agents | API Testing | CI/CD | Cloud NativeHybrid workSenior-level Full TimeBengaluru, INDIA, India7h ago
-
IT Support Specialist 1 (Regional) INR 2000K-2000KAgile | Apache Spark | Azure Data | Azure Data Factory | Azure DevOpsSenior-level Full TimeBengaluru, KA, IN7h ago
-
Software Engineer I - Python, AWS INR 2200K-3600KAWS | AWS IAM | Agile | Amazon CloudWatch | Amazon EMRSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
Senior Data Engineer - Azure Databricks INR 300K-500KACID | Apache Spark | Azure | Azure Data | Azure Data FactorySenior-level Full TimeHyderabad, TS, India8h ago
-
Software Engineer III - PySpark, ETL, AWS INR 2000K-2450KAWS | AWS EMR | Agile methodologies | Apache Spark | Application ResiliencySenior-level Full TimeHyderabad, Telangana, India10h ago
-
Sr. Systems Engineer (3 - 6 years, Kafka, IBM MQ, DevOps, Cloud, Gen AI, Automation) INR 2535K-4000KApache Kafka | KSQLDB | Kafka | Kafka Connect | Kafka StreamsHybrid workSenior-level Full TimeBengaluru, INDIA, India10h ago
-
Lead Software Engineer - KDB Developer INR 2500K-3900KAgile | Automation | Cloud Computing | Continuous Delivery | JavaSenior-level Full TimeMumbai, Maharashtra, India11h ago
-
Software Engineer (6months to 2 Years of experience - JAVA/Python Backend, CICD, APIs, GEN AI, Microservices, Cloud) INR 1430K-2000KAgile | Apache Kafka | Automated testing | C++ | CI/CDContinued learning opportunities | MentorshipEntry-level Full TimeBengaluru, INDIA, India11h ago
-
Data Scientist lead - Applied AI/ML- Agentic/Gen AI, Python INR 2100K-2600KAPIs | AWS | Agentic AI | Azure | ChatbotsSenior-level Full TimeBengaluru, Karnataka, India12h ago
-
Data Engineer INR 1500K-1800KAWS | Amazon Kinesis | Apache Airflow | Apache Beam | Apache FlinkGlobal induction program | Talent development programs | Time off for digital disconnect days | Time off for volunteering | Wellbeing programsMid-level Full TimeBengaluru, KA, India13h ago
-
Apache Airflow | Apache Hadoop | Apache Spark | App Service | Azure AppFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeKolkata DN 57, India16h ago
-
Apache Airflow | Apache Spark | Azure Cosmos | Azure Cosmos DB | Azure DataSenior-level Full TimeKolkata DN 57, India16h ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache Spark | App ServiceSenior-level Full TimeKolkata DN 57, India16h ago
-
API Integration | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India16h ago
-
Amazon Web Services | Apache Airflow | Apache Databricks | Apache Hadoop | Apache SparkSenior-level Full TimeKolkata DN 57, India16h ago
-
API ingestion | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India16h ago