Data Engineer - WkStrm3 - Syslog-NG to Iceberg
Tasks
- Align log schemas to Iceberg table definitions
- Build Confluent Kafka to Iceberg ingestion using supported APIs
- Collaborate on Iceberg catalog configuration and partition strategies
- Configure syslog ng to write JSON logs to rolling files
- Design streaming ingestion pipelines
- Handle parsing errors with dead letter queue
- Implement syslog ng to Iceberg batch ingestion pipelines
- Ingest data into Apache Iceberg tables
- Parse JSON logs with Spark file source streaming
- Present pipeline results in stakeholder demo
- Recover from checkpoint corruption
- Review AI generated PySpark code for security and performance
- Set up alerting on parse failures
- Write unit tests for field mapping and ingestion idempotency
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Apache Kafka | Apache Spark | Azure Blob | Azure Blob Storage | Blob Storage | Dead Letter Queue | GitHub Copilot | HDFS | JSON | Linux | PII Tokenization | Protegrity | PySpark | RFC5424 | S3 | Structured Streaming | Syslog-ng | Unit Testing
Education
N/A
Related jobs
-
Apache Airflow | Apache Hadoop | Apache Spark | App Service | Azure AppFlexibility programmes | Inclusive benefits | MentorshipSenior-level Full TimeKolkata DN 57, India-1d ago
-
Apache Airflow | Apache Spark | Azure Cosmos | Azure Cosmos DB | Azure DataSenior-level Full TimeKolkata DN 57, India-1d ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache Spark | App ServiceSenior-level Full TimeKolkata DN 57, India-1d ago
-
API Integration | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India-1d ago
-
Amazon Web Services | Apache Airflow | Apache Databricks | Apache Hadoop | Apache SparkSenior-level Full TimeKolkata DN 57, India-1d ago
-
API ingestion | Agile | Apache Airflow | Apache Hadoop | Apache KafkaFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeKolkata DN 57, India-1d ago
-
Ab Initio | Apache Airflow | BigQuery | Cloud Storage | Cloud platformFlexible work arrangements | Inclusive workplace culture | MentorshipSenior-level Full TimeBengaluru Millenia, India-1d ago
-
Ab Initio | BigQuery | Cloud Composer | Cloud Storage | Cloud platformFlexible work programs | Inclusive workplace culture | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India-1d ago
-
Ab Initio | Apache Airflow | BigQuery | Cloud Storage | Cloud platformFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India-1d ago
-
Ab Initio | Apache Dataflow | BigQuery | Cloud Composer | Cloud StorageCompetitive compensation | Flexibility programmes | Hybrid work environment | Inclusive benefits | MentorshipSenior-level Full TimeBengaluru Millenia, India-1d ago
-
AWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon EMRAI certifications | Ethical AI focus | Mentorship | World-class trainingSenior-level Full TimeIndia-Hyderabad9h ago
-
Mid-level Full TimePune, Maharashtra, India11h ago
-
Senior-level Full TimeBangalore, Karnataka, India12h ago
-
Senior-level Full TimeBangalore, Karnataka, India12h ago
-
AWS | AWS Lambda | Agile | Apache Spark | Data ModelingSenior-level Full TimeBengaluru, Karnataka, India13h ago
-
Software Engineer (6months to 2 Years of experience - JAVA/Python Backend, CICD, APIs, GEN AI, Microservices, Cloud) INR 2000K-2156KAgile | Algorithms | Apache Kafka | Automated testing | C++Continued learning opportunities | Hybrid work arrangement | MentorshipEntry-level Full TimeBengaluru, INDIA, India14h ago
-
Senior-level Full TimeBengaluru, INDIA, India15h ago
-
Software Engineer III - Java, AWS INR 1800K-2200KAWS | Apache Iceberg | Apache Spark | Authentication | AuthorizationSenior-level Full TimeBengaluru, Karnataka, India15h ago
-
Entry-level Full TimeBangalore16h ago
-
Senior Automation Engineer INR 2156K-2647KALB | API Gateway | AWS Lambda | Agile | Amazon CloudWatchSenior-level Full TimeHyderabad, India17h ago
-
Sr Data Engineer INR 3200K-4600KAWS | Apache Airflow | Apache Spark | Data Modeling | Google CloudCommuter benefits | Disability coverage | Financial wellness support | Healthcare | Life insuranceSenior-level Full TimeBengaluru, India17h ago
-
Senior-level Full Timebengaluru, India17h ago
-
Principal Engineer – Data Platform INR 2000K-2000KAWS | Access Control | Apache Flink | Apache Iceberg | Apache KafkaContinuous learning | Culture first work environment | High-trust work environment | Unlimited vacation policySenior-level Full TimeBengaluru17h ago
-
Systems Integration Specialist Advisor INR 2516K-3356KAWS | Azure | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeBangalore, KA, IN22h ago
-
Data Engineer (PySpark) INR 1500K-2500KAI Assisted Development | Data Mapping | Data Modeling | Data Warehousing | Data profilingMid-level Full TimeBengaluru, Karnataka, India22h ago