Data Engineer - AWS/Databricks
Reston, VA, United States
R
USD 170K-243K (estimate) Senior-level Full Time
Tasks
- Build scalable PySpark data pipelines in Databricks
- Deploy and version data engineering assets with Git and CI CD
- Design Delta Lake tables
- Develop ETL and ELT workflows
- Implement cloud native data solutions on AWS
- Implement data governance data quality validation lineage tracking
- Implement data transformations with Spark SQL and DataFrame APIs
- Integrate source systems into centralized data warehouse
- Monitor pipeline health and cluster utilization
- Optimize pipeline performance and query tuning
- Perform technical discovery and legacy system mapping
- Support ad hoc data access and reusable data assets
Perks/Benefits
Skills/Tech-stack
ACID | AWS | AWS CloudWatch | AWS Glue | Adaptive Query | Adaptive query execution | Amazon RDS | Amazon S3 | Apache Spark | Broadcast Join | CI/CD | Caching | Data Governance | Data Lineage | Data Quality | DataFrame | Databricks | Delta Lake | ELT | ETL | Git | GitLab | IAM | Jenkins | Partitioning | PySpark | Python | Query Execution | Schema Enforcement | Spark SQL
Education
Roles
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R6d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R19d ago
-
Staff Data Scientist - Digital Intelligence USD 191K-230KAdversarial Modeling | Anomaly Detection | Apache Spark | Data Processing | Distributed dataSenior-level Full TimeHybrid - US R7h ago
-
Data Scientist ll - Digital Intelligence USD 140K-170KDatabricks | Experiment analysis | Feature Engineering | Machine Learning | Model EvaluationMid-level Full TimeHybrid - US R8h ago
-
Data Engineer USD 120K-150KAPIs | Access Control | Airbyte | Amazon Redshift | Apache AirflowFully remote | Work independentlyMid-level Full TimeUnited States (Remote) R13h ago
-
Senior Software Engineer (Applied AI) USD 160K-200KAPIs | AWS | Agentic Workflows | Best practices | Cost monitoringCareer development | Discretionary performance bonus | Equity options | Health benefitsSenior-level Full TimeSeattle, New York City, Boston, or … R14h ago
-
Staff Data Platform Engineer USD 213K-245KAWS | AWS Lambda | CDK | CI/CD | Code Quality401k match | Catered lunches | Dental insurance | Flexible work schedule | HackathonsSenior-level Full TimeRemote, United States R15h ago
-
Senior Software Engineer, Perception Architecture USD 184K-296KC++ | Computer Vision | Data pipeline | Deep learning | Machine LearningSenior-level Full TimePittsburgh, PA or Remote R15h ago
-
Senior Software Engineer, Data & AI Platform USD 171K-224KApache Spark | Batch Processing | CI/CD | Cloud Native | Data Engineering401k retirement savings plan | Annual cash bonus | Equity grants | Flexible time off | Generous parental leaveSenior-level Full TimeRemote, USA R16h ago
-
Staff Machine Learning Engineer USD 118K-136KCI/CD | Data Governance | Data Modeling | Data Pipelines | Data Quality401k match | Dental insurance | Family planning resources | Flexible vacation policy | Fully remoteSenior-level Full TimeRemote - USA R16h ago
-
Associate Machine Learning AI Engineer USD 100K-138KAPI Design | API Key | API key management | AWS | Agent workflowsHealth insurance | Remote work option | Unlimited Paid Days OffMid-level Full TimeRemote - United States R17h ago
-
Senior Analytics Engineer USD 115K-193KAirflow | Amazon Athena | Apache Iceberg | BigQuery | CI/CD401k | Dental insurance | Health insurance | Paid time off | Remote work opportunitySenior-level Full TimeRemote (United States | Canada) R19h ago
-
Autonomy | C++ | DDS | Edge Computing | Embedded Systems401k | Healthcare coverage | PTO | Paid Holidays | Phone stipendMid-level Full TimeSan Carlos - Hybrid R19h ago
-
A/B | A/B Testing | AWS | AWS Bedrock | Anthropic401k | Health insurance | Paid time off | Stipends | Wellness benefitsSenior-level Full TimeSan Carlos - Hybrid R19h ago
-
A/B | A/B Testing | AWS CDK | AWS CodeBuild | AWS CodePipeline401k plan | Healthcare coverage | Paid time off | Phone stipend | Wellness stipendSenior-level Full TimeSan Carlos - Hybrid R19h ago
-
Autonomy | C++ | CPU GPU | CPU GPU Management | DDS401k | Healthcare benefits | Paid Holidays | Paid time off | Phone stipendSenior-level Full TimeSan Carlos - Hybrid R19h ago
-
Senior Data Engineer USD 108K-160KAPI | AWS Lambda | Airflow | Algolia | Amplitude401k | Dental insurance | Disability insurance | Life insurance | Medical insuranceSenior-level Full TimeRemote US R20h ago
-
Principal Software Engineer USD 105K-159KAWS | Agile | Azure | Bitbucket | CI/CDFlexible hybrid work model | Health and life insurance | Paid time off | Pension/retirement benefits | Personal/Family Care leavesSenior-level Full TimeTampa, FL, United States R23h ago
-
Advanced Analytics | Claims data | Claims data analysis | Dashboarding | Data AnalysisMid-level Full TimeBoston, Massachusetts, United States; Remote, United … R1d ago
-
Principal AI Engineering Architect USD 224K-308KAWS | Data Processing | Docker | Google Cloud | High AvailabilityAdoption and surrogacy reimbursement | Cancer Care Program | Employee Assistance Program (EAP) | Fertility coverage | Flexible time offSenior-level Full TimeUnited States - Remote R1d ago
-
Staff Data Analytics Engineer USD 159K-220KApache Airflow | Apache Spark | BigQuery | DBT | Data ContractsFlexible working hours | Remote work flexibilitySenior-level Full TimeAmericas - Remote, US - Remote, … R1d ago
-
Analytics Engineer USD 86K-165KAPI | Agile | Data Cleansing | Data Modeling | Data orchestration401k match | Dental insurance | Flexible work schedules | Life insurance | Medical insuranceSenior-level Full TimeUS-FL-REMOTE, United States R1d ago
-
Data Scientist Lead Consultant - Remote USD 90K-153KAWS | Azure | Machine Learning | Predictive Modeling | PythonConnectivity reimbursement | Equipment provided | Remote workSenior-level Full TimeUSA - NJ (Remote), United States R1d ago
-
Principal Data Engineering - Hybrid USD 112K-193KAPI Design | Apache Spark | Artifactory | Azure | Azure Databricks401k contribution | Career development opportunities | Comprehensive benefits package | Hybrid work schedule | Incentive and recognition programsSenior-level Full TimePrimary location: Eden Prairie, Minnesota R1d ago
-
Technical Director - ML & Algorithms USD 161K-225KAnomaly Detection | Apache Flink | Apache Spark | Auditability | Classification401k matching | Certificate reimbursement | Disability coverage | Health and wellness coverage | Leadership trainingExecutive-level Full TimeUnited States, Remote, United States R1d ago