Data Engineer - WkStrm3 - Syslog-NG to Iceberg
Tasks
- Align schemas between syslog ng output and Iceberg tables
- Build PySpark Structured Streaming jobs
- Collaborate on Iceberg catalog configuration and partitioning
- Configure syslog ng JSON logging to rolling files
- Design streaming ingestion pipelines
- Implement error handling with dead letter queues
- Ingest data into Apache Iceberg tables
- Present streaming pipeline results in stakeholder demos
- Read and parse log files incrementally into Iceberg
- Recover from checkpoint corruption
- Review and refine PySpark pipelines for standards security and performance
- Scaffold iterate test and document streaming code
- Set up alerting for parse failures
Perks/Benefits
- N/A
Skills/Tech-stack
Amazon S3 | Apache Iceberg | Apache Kafka | Azure Blob | Azure Blob Storage | Blob Storage | Confluent Kafka | Dead Letter Queue | GitHub Copilot | HDFS | JSON | PII Tokenization | Protegrity | PySpark | RFC5424 | Spark File Source | Structured Streaming | Syslog-ng | Unit Testing
Education
N/A
Related jobs
-
Senior-level Full TimeIN-TN-Chennai8h ago
-
AWS Data Engineer INR 1500K-2040KAWS CDK | AWS Step Functions | Amazon Glue | Amazon RDS | Amazon RedshiftSenior-level Full TimeIN-TN-Chennai8h ago
-
AWS Data Engineer INR 1500K-2040KAWS CDK | AWS Step Functions | Amazon Glue | Amazon RDS | Amazon RedshiftSenior-level Full TimeIN-TN-Chennai8h ago
-
Mid-level Full TimeHyderabad, TS, IN; Bengaluru, KA, IN11h ago
-
Senior-level Full Timehosur road bangalore, India11h ago
-
Senior-level Full Timehosur road bangalore, India11h ago
-
Senior - Data and App Modernization INR 2500K-3500KCI/CD | Data Cleansing | Data Denormalization | Data Modeling | Data NormalizationSenior-level Full TimeGurgaon, Haryana, India13h ago
-
Data Engineer INR 1500K-2000KAWS | Ab Initio | Apache Flink | Apache Spark | Apache Spark StreamingMid-level Full TimeBengaluru, Karnataka, India14h ago
-
DE&A - Snowflake - Specialist INR 1000K-2000KAgile | Amazon S3 | CI/CD | Data Migration | Data ModelingMid-level Full TimeBengaluru South, Karnataka, India15h ago
-
Senior Backend Engineer (Big Data) INR 2516K-3380KAmazon Web Services | Apache Kafka | Apache Spark | BigQuery | Cloud ComputingAccident insurance | Disability insurance | Employee assistance program | Flexible paid time off | Life insuranceSenior-level Full TimeHyderabad, India21h ago
-
Data Engineer - Specialist INR 1500K-2146KAWS | AWS Glue | Amazon Kinesis | Amazon S3 | Apache AirflowEmployee assistance programme | Flexible schedule | Health insurance | Parental leave | Professional development opportunitiesSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …21h ago
-
Data Engineer – Specialist INR 1628K-2146KAWS | AWS Glue | Amazon Kinesis | Amazon S3 | Apache AirflowEmployee assistance programme | Flexible schedules | Health insurance | Holiday purchase scheme | Parental leaveSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …21h ago
-
Senior Data Engineer INR 2000K-2146KAWS Glue | AWS S3 | Airflow | Amazon Athena | Amazon RedshiftEmployee assistance programme | Flexible schedules | Health insurance | Holiday purchase scheme | Parental leaveSenior-level Full TimeBuilding No 12D, Floor 5, Raheja …21h ago
-
Senior Data Engineer INR 2000K-2146KAWS | AWS Glue | Amazon Athena | Amazon MWAA | Amazon RedshiftEmployee assistance programme | Flexible schedules | Health insurance | Parental leave | Professional development opportunitiesSenior-level Full TimeBuilding No 12D, Floor 5, Raheja …21h ago
-
Senior Engineer, AI/Machine Learning INR 2500K-4500KAWS | Agile | Apache Airflow | Apache Kafka | C++Senior-level Full TimeIndia-Pune21h ago
-
Senior Engineer, AI & Machine Learning INR 2500K-4500KAWS | Agent Studio | Agile | Airflow | Apache KafkaSenior-level Full TimeIndia-Pune21h ago
-
Data Engineer INR 2000K-2146KAWS | Amazon Redshift | Amazon Web Services | Apache Spark | BigQueryEmployee assistance programme | Health insurance | Professional developmentSenior-level Full TimeEcospace Campus 3A, 4th Floor, Outer …21h ago
-
Data Engineer III INR 2520K-3380KAWS Kinesis | Amazon DynamoDB | Amazon S3 | Apache Airflow | Apache SparkSenior-level Full TimeDLF Downtown1, Chennai, IN, India21h ago
-
Access Control | Azure Purview | Compliance Monitoring | Data Cataloging | Data ClassificationFlexibility programmes | Inclusive benefits | Mentorship | Professional growth and learningSenior-level Full TimeBengaluru Millenia, India21h ago
-
Machine Learning Engineer I INR 2000K-2700KAPI Development | AWS IAM | Airflow | Amazon EC2 | Amazon EMRMid-level Full TimeDLF Downtown1, Chennai, IN, India21h ago
-
Access Control | Azure Databricks | Azure Purview | Compliance Monitoring | Data Access ControlSenior-level Full TimeHyderabad, India21h ago
-
Agile | Apache Airflow | Apache Hadoop | Azure Data | Azure Data FactoryFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBhubaneswar - Ihub, India21h ago
-
Access Controls | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageFlexibility programmes | Inclusive benefits | Mentorship | Professional growth and learningSenior-level Full TimeHyderabad, India21h ago
-
Access Control | Azure Databricks | Azure Purview | Compliance Monitoring | Data Access ControlSenior-level Full TimeHyderabad, India21h ago
-
IN_Manager_Data Engineer_Application Technology_Advisory_Kolkata INR 1500K-2000KAsset bundles | Azure Cosmos | Azure Cosmos DB | Azure Data | Azure Data FactoryFlexible work programs | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeKolkata DN 57, India21h ago