Data Engineer - Streaming (WkStream 2 - Kafka)
Tasks
- Apply PII detection and Protegrity tokenization hooks
- Build PySpark Structured Streaming applications
- Configure Kafka source parameters
- Design streaming ingestion pipelines
- Handle errors with dead letter queues
- Implement foreachBatch micro batch writes
- Ingest data from Kafka topics
- Map schemas and write to Apache Iceberg
- Parse JSON and Avro payloads
- Recover from checkpoint failures
- Set up checkpoint and offset management
- Support UAT and review compliance
- Validate schema and row counts
- Verify Kafka offset commits
- Write unit and integration tests
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Avro | Apache Iceberg | Apache Kafka | Apache Spark | Checkpoints | Confluent Kafka | Dead Letter Queue | ForeachBatch | JSON | Java | Kafka Connect | PySpark | S3 | Scala | Structured Streaming | Watermarking
Education
N/A
Roles
Related jobs
-
Senior-level Full TimeIN-TN-Chennai6h ago
-
AWS Data Engineer INR 1500K-2040KAWS CDK | AWS Step Functions | Amazon Glue | Amazon RDS | Amazon RedshiftSenior-level Full TimeIN-TN-Chennai6h ago
-
AWS Data Engineer INR 1500K-2040KAWS CDK | AWS Step Functions | Amazon Glue | Amazon RDS | Amazon RedshiftSenior-level Full TimeIN-TN-Chennai6h ago
-
Forward Deployed Engineer, GenAI, Google Cloud INR 2500K-5000KAlgorithms | Apache Beam | Apache Hadoop | Apache Spark | C++Embedded customer development | Technical feedback to product roadmap | White glove deploymentSenior-level Full TimeBengaluru, Karnataka, India; Gurugram, Haryana, India8h ago
-
Mid-level Full TimeHyderabad, TS, IN; Bengaluru, KA, IN9h ago
-
AI Engineer INR 1500K-2400KAgentic AI | Agile | Apache Spark | Artificial Intelligence | AutomationAccidental insurance | Life insurance | Meal arrangements | Medical insurance | Professional development opportunitiesMid-level Full TimeGurugram, HR, IN10h ago
-
DE&A - Core - Big Data Engineering - DBT INR 2800K-4000KApache Maven | Apache Spark | Functional Programming | Higher-order functions | ImmutabilitySenior-level Full TimeIndia10h ago
-
Senior - Data and App Modernization INR 2500K-3500KCI/CD | Data Cleansing | Data Denormalization | Data Modeling | Data NormalizationSenior-level Full TimeGurgaon, Haryana, India11h ago
-
Data Engineer INR 1500K-2000KAWS | Ab Initio | Apache Flink | Apache Spark | Apache Spark StreamingMid-level Full TimeBengaluru, Karnataka, India13h ago
-
DE&A - Snowflake - Specialist INR 1000K-2000KAgile | Amazon S3 | CI/CD | Data Migration | Data ModelingMid-level Full TimeBengaluru South, Karnataka, India13h ago
-
Site Reliability Engineer INR 1170K-1500KAlerting | Ansible | Automation | Bash | ChefHealthcare coverage | Hybrid work | Mentorship | Online learning platform | Paid time offEntry-level Full TimeIND-Trivandrum-Equifax Analytics-PEC, India19h ago
-
Senior Backend Engineer (Big Data) INR 2516K-3380KAmazon Web Services | Apache Kafka | Apache Spark | BigQuery | Cloud ComputingAccident insurance | Disability insurance | Employee assistance program | Flexible paid time off | Life insuranceSenior-level Full TimeHyderabad, India19h ago
-
Data Engineer - Specialist INR 1500K-2146KAWS | AWS Glue | Amazon Kinesis | Amazon S3 | Apache AirflowEmployee assistance programme | Flexible schedule | Health insurance | Parental leave | Professional development opportunitiesSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …19h ago
-
Data Engineer – Specialist INR 1628K-2146KAWS | AWS Glue | Amazon Kinesis | Amazon S3 | Apache AirflowEmployee assistance programme | Flexible schedules | Health insurance | Holiday purchase scheme | Parental leaveSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …19h ago
-
Senior Data Engineer INR 2000K-2146KAWS Glue | AWS S3 | Airflow | Amazon Athena | Amazon RedshiftEmployee assistance programme | Flexible schedules | Health insurance | Holiday purchase scheme | Parental leaveSenior-level Full TimeBuilding No 12D, Floor 5, Raheja …19h ago
-
Senior Data Engineer INR 2000K-2146KAWS | AWS Glue | Amazon Athena | Amazon MWAA | Amazon RedshiftEmployee assistance programme | Flexible schedules | Health insurance | Parental leave | Professional development opportunitiesSenior-level Full TimeBuilding No 12D, Floor 5, Raheja …19h ago
-
Junior Data Engineer INR 300K-410KAWS | Agile | Azure Event | Azure Event Hubs | CI/CDCollaborative culture | Professional growth and developmentEntry-level Full TimeGCC, India19h ago
-
Senior Engineer, AI/Machine Learning INR 2500K-4500KAWS | Agile | Apache Airflow | Apache Kafka | C++Senior-level Full TimeIndia-Pune19h ago
-
Senior Engineer, AI & Machine Learning INR 2500K-4500KAWS | Agent Studio | Agile | Airflow | Apache KafkaSenior-level Full TimeIndia-Pune19h ago
-
Data Engineer INR 2000K-2146KAWS | Amazon Redshift | Amazon Web Services | Apache Spark | BigQueryEmployee assistance programme | Health insurance | Professional developmentSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …19h ago
-
AI Engineer INR 2000K-3500KC++ | CUDA | CUDNN | Calculus | Entity recognitionHybrid work | Professional developmentSenior-level Full TimeGurugram - Good Earth, India R19h ago
-
Data Engineer III INR 2520K-3380KAWS Kinesis | Amazon DynamoDB | Amazon S3 | Apache Airflow | Apache SparkSenior-level Full TimeDLF Downtown1, Chennai, IN, India19h ago
-
Access Control | Azure Purview | Compliance Monitoring | Data Cataloging | Data ClassificationFlexibility programmes | Inclusive benefits | Mentorship | Professional growth and learningSenior-level Full TimeBengaluru Millenia, India19h ago
-
Machine Learning Engineer I INR 2000K-2700KAPI Development | AWS IAM | Airflow | Amazon EC2 | Amazon EMRMid-level Full TimeDLF Downtown1, Chennai, IN, India19h ago
-
Access Control | Azure Databricks | Azure Purview | Compliance Monitoring | Data Access ControlSenior-level Full TimeHyderabad, India19h ago