Senior Data Engineer, DPD Team (Remote, International)
Tasks
- Build deploy data tools and datasets
- Create Kafka consumers using Spark Streaming
- Create maintain monitor data flows
- Design build maintain scalable distributed data processing systems
- Evaluate data tools and technologies
- Ingest validate process internal third party data
- Maintain enhance Spark and Hive aggregation jobs framework
- Manage backups, retention, high availability, and capacity planning
- Monitor data quality across systems
- Optimize data jobs using Kafka Hadoop Presto Spark Kubernetes
- Provide mentorship and guidance to junior team members
- Review and approve DDL for databases and data processing jobs
Perks/Benefits
Skills/Tech-stack
Apache Airflow | Apache Hive | Apache Iceberg | Apache Kafka | Apache Spark | Cloud platform | Distributed Systems | Docker | Google BigQuery | Google Cloud | Google Cloud Platform | Graphite | Hadoop | High Availability | Hive | Java | Kubernetes | Linux | Looker | Presto | Python | Query Optimization | RDBMS | SQL | Scala | Spark Streaming | Trino
Education
N/A
Roles
Related jobs
-
AI Gateways | AWS CDK | Chunking | Context engineering | Cost Tracking401k match | Counseling membership | Flexible time away | Life insurance | Long-term disabilityMid-level Full Time-REMOTE, USA- R14h ago
-
Agile | Apache Airflow | Artificial Intelligence | Automated testing | BigQueryCollaborative culture | Flexible working hours | Performance evaluations | Professional development opportunities | Remote workSenior-level Full TimeIdaho R21h ago
-
Agile | Apache Airflow | BigQuery | CI/CD | Cloud StorageCollaborative company culture | Flexible working hours | Professional development opportunities | Remote-first work environmentSenior-level Full TimeMinnesota R21h ago
-
Agile | Airflow | BigQuery | CI/CD | Cloud StorageCollaborative culture | Professional development | Remote-first flexibilitySenior-level Full TimeColorado R21h ago
-
Agile | Airflow | BigQuery | CI/CD | Cloud StorageCollaborative company culture | Flexible working hours | Professional development opportunities | Remote-first environmentSenior-level Full TimeColumbia R21h ago
-
Agile | Airflow | Automated testing | BI tools | BigQueryCollaborative company culture | Professional development | Remote-first flexible hoursSenior-level Full TimeIllinois R21h ago
-
Agile | Airflow | BigQuery | CI/CD | Cloud StorageCollaborative company culture | Professional development opportunities | Remote-first flexible hoursSenior-level Full TimeFlorida R21h ago
-
Agile | Airflow | Automated testing | BI tools | BigQueryCollaborative culture | Flexible working hours | Professional development | Remote-first environmentSenior-level Full TimeCalifornia R21h ago
-
Agile | Apache Airflow | Automated testing | BigQuery | CI/CDCollaborative company culture | Professional development opportunities | Remote-first flexibilitySenior-level Full TimeConnecticut R21h ago
-
Agile | Airflow | BI tools | BigQuery | CI/CDCollaborative & Innovative Culture | Professional development | Remote-first flexible hoursSenior-level Full TimeArizona R21h ago
-
Data Integrity | Data Mapping | Data Migration | Data Transformation | Data Validation401k | Career development | Certifications and training | Dental insurance | HolidaysSenior-level Full TimeWashington, District of Columbia, United States … R1d ago
-
Defensive Security AI Scientist USD 240K-260KAccelerate | Artificial Intelligence | CISA KEV | CUDA | CVSS401k plan with company matching | Bereavement | Disability insurance | Employee assistance program | Employee discount programSenior-level Full TimeRemote - Nationwide, United States R1d ago
-
ML Infrastructure Engineer USD 145K-165KAWS | Amazon Elastic Kubernetes Service | Amazon SageMaker | BigQuery | CD pipelinesHealth benefits | Paid time off | Remote work optionMid-level Full TimeBoston, MA R2d ago
-
ML Infrastructure Engineer USD 145K-165KAWS | Amazon SageMaker | BigQuery | CI/CD | CloudFormationBenefits plans | Remote work optionMid-level Full TimeNew York, New York, United States R2d ago
-
ML Infrastructure Engineer USD 145K-165KAWS | Amazon SageMaker | BigQuery | BigQuery datasets | CI/CDCompany benefits plan enrollment | Health benefits | Performance-based bonus | Remote work optionMid-level Full TimeLos Angeles, California, United States R2d ago
-
AI | AWS | DBA | Database systems | DevOpsDental insurance | Flexible working hours | Health insurance | Paid time off | Professional developmentSenior-level Full TimeMinnesota R2d ago
-
AI machine learning | AWS | Cloud platform | DBA operations | Database systemsDental insurance | Flexible working hours | Health insurance | Paid time off | Professional developmentMid-level Full TimeIllinois R2d ago
-
C# | MATLAB | NumPy | Pandas | PythonFlexible hours | Non permanent employment | Part-time project workSenior-level Full TimeNew York, New York, United States … R2d ago
-
Senior Python Developer - Code Migration Specialist USD 160K-160KBash | Coverage.py | Dagger | Docker | GcovFlexible schedule | Freelance project-based collaboration | Fully remote | Supportive global communitySenior-level Full TimeNew York, United States - Remote R2d ago
-
Senior Python Developer - Code Migration Specialist USD 160K-160KBash | Black box testing | Black-box | Box testing | Code CoverageFlexible schedule | Project-based collaboration | Remote work | Supportive global communitySenior-level Full TimeTexas, United States - Remote R2d ago
-
Senior Python Developer - Code Migration Specialist USD 160K-160KBash | Black box testing | Black-box | Box testing | Code CoverageFlexible hours | Fully remote | Project-based collaboration | Supportive communitySenior-level Full TimeMichigan, United States - Remote R2d ago
-
Senior Python Developer - Code Migration Specialist USD 160K-160KBash | Black box testing | Black-box | Box testing | Coverage.pyFlexible schedule | Freelance project-based collaboration | Fully remote | Project based hours | Supportive global communitySenior-level Full TimeFlorida, United States - Remote R2d ago
-
Senior Python Developer - Code Migration Specialist USD 160K-160KBash | Coverage.py | Dagger | Docker | GcovFlexible schedule | Freelance project-based collaboration | Fully remote | Opportunity to contribute to AI projects | Supportive global communitySenior-level Full TimeUnited States - Remote R2d ago
-
Senior AI Data Engineer USD 160K-200KAWS | AWS Athena | AWS Glue | AWS Lambda | Amazon Redshift401k matching | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeSan Diego, California, United States R2d ago
-
Causal Inference | Classification | Clustering | Data Warehousing | Experiment designFlexible PTO | Home office stipend | Learning budget | Paid health, dental, vision | Parental leaveSenior-level Full TimeBoston or Remote R2d ago