Data Engineer
Tasks
- Build ETL and ELT workflows
- Build vector ingestion and embedding pipelines
- Design batch and streaming pipelines
- Develop transformations and orchestration
- Handle late arriving data and schema drift
- Implement CI/CD for data pipelines
- Ingest operational data into data platform
- Maintain data quality and observability
- Manage backfills and replays
- Monitor pipeline health and resolve incidents
- Own data pipelines
- Package data products with contracts and versioning
Perks/Benefits
- N/A
Skills/Tech-stack
Airflow | Amazon Kinesis | Apache Beam | Apache Flink | Apache Hudi | Apache Kafka | Apache Kafka Connect | Apache Spark | Avro | CI/CD | DBT | Dagster | Dbt tests | Debezium | Delta Lake | Feast | Fivetran | GitOps | GoldenGate | Great Expectations | Iceberg | Integration Testing | JSON Schema | Java | Kafka | Kafka Connect | OpenLineage | PGVector | Pinecone | Prefect | Protobuf | Python | Qlik Replicate | SQL | Scala | Soda | Spark Structured Streaming | Structured Streaming | Tecton | Unit Testing
Education
N/A
Roles
Regions
Countries
States
Related jobs
-
Senior-level Full TimeLK-1-Colombo4h ago
-
AWS Cloud | AWS Cloud Services | Agile | Artificial Intelligence | Asynchronous ArchitectureCareer growth opportunities | Exposure to modern cloud native technologies | MentorshipSenior-level Full TimeSri Lanka1d ago
-
Agent Frameworks | Architecture Design | C# | CI/CD | Cloud NativeHybrid work options | Inclusive workplaceSenior-level Full TimeColombo, Western Province, Sri Lanka1d ago
-
API Gateway | AWS | AWS Well Architected | AWS Well-Architected Framework | AgileCareer growth plan | Flexible work options | Internal conferences | Knowledge sharing sessions | MentorshipSenior-level Full TimeColombo, Sri Lanka R3d ago
-
Mid-level Full TimeColombo, Sri Lanka3d ago
-
Senior-level Full TimeSL Sri Jayewardenepura Kotte, Sri Lanka3d ago
-
Senior-level Full TimeColombo, Sri Lanka4d ago
-
Mid-level Full TimeColombo, Sri Lanka4d ago
-
AWS | Access Control | Apache Spark | Audit Logging | ContainerizationCareer excellence program | Employee assistance program | Engagement events | Flexible work | Fruits and nuts Mondays and WednesdaysSenior-level Full TimeColombo, Sri Lanka4d ago
-
Associate/ Engineer - Artificial Intelligence LKR 960K-960KAgentic AI | Data Pipelines | Deep learning | Feature Engineering | Generative AIOn-site workMid-level Full TimeBattaramulla, Sri Lanka7d ago
-
Azure Fabric | Business Intelligence | Cloud services | Data Pipelines | Data TransformationInternational exposure | Medical insurance | Open work culture | Recognition appreciation | Recreational activities and eventsSenior-level Full TimeColombo, Western Province, Sri Lanka - … R8d ago
-
Azure | Azure Data | Azure Data Factory | Azure Fabric | Business IntelligenceInternational exposure | Medical insurance | Open work culture | Recreational activities | Training fundedSenior-level Full TimeColombo, Western Province, Sri Lanka - … R8d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudMid-level Full TimeColombo, Sri Lanka9d ago
-
Senior-level Full TimeColombo, Sri Lanka11d ago
-
Software Engineer LKR 960K-960KGRPC | Go | Helm | JVM | JavaFlexible working hours | Remote-friendly policiesMid-level Full TimeColombo11d ago
-
Agent Orchestration | Artificial Intelligence | Distributed Systems | Embeddings | Graph DatabasesSenior-level Full TimeColombo, Sri Lanka17d ago
-
AI Engineer LKR 720K-960KAI Agent framework | AI Foundry | AI Services | API Integration | Agent FrameworkHybrid workEntry-level Full TimeColombo, Western Province, Sri Lanka18d ago
-
AWS | Azure | CI/CD | Cloud Native | Cloud Native ArchitectureCareer growth | Flexible working environment | Inclusive culture | Remote-friendly cultureSenior-level Full TimeColombo25d ago
-
Access Control | Always On | Apache | Automation | Backup and RecoverySenior-level TemporaryColombo, Sri Lanka1mo ago
-
Associate DevOps Engineer LKR 960K-960KAmazon Web Services | Ansible | CI/CD | Cloud platform | DockerFlexible work schedule | On-site workMid-level Full TimeBattaramulla, Sri Lanka1mo ago
-
Software Engineer LKR 960K-960KDistributed Systems | GRPC | Go | Helm | JavaCareer growth | Flexible working environment | Mentorship | Remote-friendly cultureMid-level Full TimeColombo1mo ago
-
Engineer - Data Engineering LKR 960K-960KAWS Glue | AWS Lambda | Airflow | Amazon EMR | Amazon Web ServicesMid-level Full TimeSysco LABS - Sri Lanka1mo ago
-
Applied AI Engineer - IFS Loops LKR 960K-960KAI orchestration | API | Application Integration | CRM | Data FlowHybrid workMid-level Full TimeColombo, Western Province, Sri Lanka1mo ago
-
CI/CD | Cloud Computing | Containerization | Experiment tracking | Feature StoreEntry-level Full TimeColombo, LK1mo ago
-
Agile | Apache Hadoop | Apache Spark | Azure | Azure DataEntry-level Full TimeColombo, LK1mo ago