Data Engineer - WkStrm3 - Syslog-NG to Iceberg
Tasks
- Align schemas between syslog ng output and Iceberg tables
- Build PySpark Structured Streaming jobs
- Collaborate on Iceberg catalog configuration and partitioning
- Configure syslog ng JSON logging to rolling files
- Design streaming ingestion pipelines
- Implement error handling with dead letter queues
- Ingest data into Apache Iceberg tables
- Present streaming pipeline results in stakeholder demos
- Read and parse log files incrementally into Iceberg
- Recover from checkpoint corruption
- Review and refine PySpark pipelines for standards security and performance
- Scaffold iterate test and document streaming code
- Set up alerting for parse failures
Perks/Benefits
- N/A
Skills/Tech-stack
Amazon S3 | Apache Iceberg | Apache Kafka | Azure Blob | Azure Blob Storage | Blob Storage | Confluent Kafka | Dead Letter Queue | GitHub Copilot | HDFS | JSON | PII Tokenization | Protegrity | PySpark | RFC5424 | Spark File Source | Structured Streaming | Syslog-ng | Unit Testing
Education
N/A
Related jobs
-
AWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon EMRAI certifications | Ethical AI focus | Mentorship | World-class trainingSenior-level Full TimeIndia-Hyderabad7h ago
-
Mid-level Full TimePune, Maharashtra, India10h ago
-
Senior-level Full TimeBangalore, Karnataka, India11h ago
-
Senior-level Full TimeBangalore, Karnataka, India11h ago
-
Software Engineer (6months to 2 Years of experience - JAVA/Python Backend, CICD, APIs, GEN AI, Microservices, Cloud) INR 2000K-2156KAgile | Algorithms | Apache Kafka | Automated testing | C++Continued learning opportunities | Hybrid work arrangement | MentorshipEntry-level Full TimeBengaluru, INDIA, India13h ago
-
Senior-level Full TimeBengaluru, INDIA, India14h ago
-
Software Engineer III - Java, AWS INR 1800K-2200KAWS | Apache Iceberg | Apache Spark | Authentication | AuthorizationSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
Principal Engineer – Data Platform INR 2000K-2000KAWS | Access Control | Apache Flink | Apache Iceberg | Apache KafkaContinuous learning | Culture first work environment | High-trust work environment | Unlimited vacation policySenior-level Full TimeBengaluru16h ago
-
Systems Integration Specialist Advisor INR 2516K-3356KAWS | Azure | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeBangalore, KA, IN21h ago
-
Data Engineer (PySpark) INR 1500K-2500KAI Assisted Development | Data Mapping | Data Modeling | Data Warehousing | Data profilingMid-level Full TimeBengaluru, Karnataka, India21h ago
-
Data Engineer - Real-Time Streaming INR 1400K-1500KApache Flink | Apache Kafka | CI/CD | Cloudera | Distributed SystemsMid-level Full TimeChennai, Tamil Nadu, India21h ago
-
Data Engineer (PySpark) INR 1500K-2000KData Compliance | Data Governance | Data Mapping | Data Pipeline Architecture | Data ProcessingMid-level Full TimeBengaluru, Karnataka, India21h ago
-
Data Engineer - Real Time Streaming INR 1800K-2500KApache Flink | Apache Kafka | CI/CD | Cloudera | Design PatternsMid-level Full TimeBengaluru, Karnataka, India21h ago
-
Senior Data Engineer INR 2520K-3380KAPI Authentication | Continuous integration | DevOps | Docker | DruidSenior-level Full TimeChennai, TN, IN21h ago
-
Data Engineer (PySpark/Informatica BDM) INR 500K-575KAWS | Azure | CI/CD | Data Governance | Data MappingSenior-level Full TimeBengaluru, Karnataka, India21h ago
-
Technology Engineer - Kafka & Streaming Platform INR 2000K-2245KAWS | Apache Kafka | Argo CD | Azure | CI/CDSenior-level Full TimeChennai, Tamil Nadu, India21h ago
-
Technology Engineer - Redis & Streaming Platforms INR 1837K-2000KAWS | Ansible | Apache Kafka | ArgoCD | AzureSenior-level Full TimeChennai, Tamil Nadu, India21h ago
-
Senior-level Full TimeBangalore, India21h ago
-
Software Principal Engineer - Datapath INR 2500K-3285KAuthentication | Authorization | BSD | CI/CD | Chaos EngineeringSenior-level Full TimeBangalore, India21h ago
-
Software Senior Principal Engineer Datapath INR 2500K-3500KAmazon Web Services | Authentication | Authorization | BSD | C#Senior-level Full TimeBangalore, India21h ago
-
AWS | Azure | CI/CD | Concurrency | DVCBest in class amenities and workspaces | Competitive health insurance coverage | Comprehensive rewards and recognition | Health and medical insurance | Holistic well-beingSenior-level Full TimeIND-KA-Bengaluru, Bagmane Solarium City, India21h ago
-
AI Engineer (Gen AI, Python, C#, JavaScript, Angular) INR 3200K-4600K.NET | API Gateway | AWS Lambda | Amazon API | Amazon API GatewayContinuing education program | Continuous learning | Family-friendly perks | Financial wellness programs | Flexible time offSenior-level Full TimeIN - HYDERABAD SKYVIEW, India21h ago
-
AWS | Agile | Azure | Bamboo | Code reviewFlexible work culture | Health insurance | Learning and development | Rewards and recognition | Wellness programsEntry-level Full TimeIND-KA-Bengaluru, Bagmane Solarium City, India21h ago
-
Data Engineer 1 INR 1400K-2000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Entry-level Full TimeIND-CORP Bengaluru-TDIM - PTT, India21h ago
-
Data Engineer I INR 1000K-1500KAzure Data | Azure Data Factory | Azure DevOps | CI/CD | Data FactoryMid-level Full TimeBengaluru, Karnataka, India21h ago