Data Engineer
Tasks
- Build Apache Spark data enrichment pipelines
- Build and scale data pipelines
- Collaborate with product, engineering, and data science teams
- Create and manage AWS Glue catalogs
- Define and enforce data contracts and schemas
- Design and maintain data architectures on AWS
- Execute data governance with lineage and schema design
- Implement data quality validation checks
- Implement observability and debugging mechanisms
- Monitor pipeline accuracy and completeness
- Optimize pipeline query and storage performance
- Orchestrate workflows with Apache Airflow
- Query data using Amazon Athena
- Troubleshoot data delivery and device tracking issues
Perks/Benefits
Skills/Tech-stack
AWS Glue | AWS S3 | Amazon Athena | Amazon Web Services | Apache Airflow | Apache Spark | Attribution | Data Contracts | Data Governance | Data Monitoring | Data Quality | ELT | ETL | GRPC | Java | Kafka | PySpark | Python | Query Optimization | REST | SQL | Streaming | Web Services
Education
Roles
Regions
Countries
States
Related jobs
-
Bash | Data Ingestion | Data Processing | Docker | GCPAsynchronous work culture | Collaborative team culture | Friendly work environment | Opportunity to build company impact | Work on inclusive learning productMid-level Full TimeKathmandu, Nepal2d ago
-
Senior-level ContractKathmandu12d ago
-
Senior Data Engineer USD 165K-216KAPI Integration | AWS Fargate | AWS S3 | Airbyte | Amazon ECSComprehensive medical cover | Group life insurance | Home working | Office snacks and lunch | Personal development and growth opportunitiesSenior-level Full TimeKathmandu, Bagmati Province, Nepal1mo ago