Data Engineer (Databricks), Assistant Vice President
Quincy, Massachusetts, United States
USD 110K-177K Executive-level Full Time
Tasks
- Analyze logs to resolve production issues
- Apply dimensional modeling and partitioning
- Automate Databricks asset deployment with Repos and CLI
- Automate processes with scripting
- Build and manage Databricks jobs workflows and notebooks
- Collaborate with IAM and security teams
- Collaborate with legal security compliance and data teams
- Containerize data workloads with Docker
- Continuously improve performance scalability and cost efficiency
- Design and maintain CI/CD pipelines
- Design and optimize Lakehouse architectures
- Design, build, and maintain scalable data pipelines
- Develop and optimize ETL ELT workflows on Databricks
- Develop reusable frameworks for data ingestion processing and orchestration
- Enable data exposure via APIs connectors and curated datasets
- Ensure adherence to data privacy and regulatory requirements
- Ensure data quality with unit testing and validation checks
- Implement data access controls and classification
- Implement data governance with Unity Catalog and AWS controls
- Implement lakehouse architecture with Bronze Silver Gold layers
- Integrate Databricks pipelines with Power Platform
- Integrate data from SQL Server and Oracle
- Maintain data lineage consistency and audit readiness
- Maintain documentation with architecture data flows and runbooks
- Monitor and troubleshoot distributed Spark workloads
- Monitor schedule and optimize Databricks workflows
- Process structured and semi-structured data
- Publish curated datasets for Power BI Power Apps and Power Automate
- Serve as Databricks and lakehouse architecture SME
- Translate business requirements into data engineering designs
- Tune Spark performance with caching and optimization
Perks/Benefits
- 401k match
- Dental insurance
- Employee assistance program
- Employee networks
- Flexible work/life support
- Health insurance
- Life insurance
- Long-term disability insurance
- Paid time off
- Paid volunteer days
- Retirement savings plan
- Vision insurance
Skills/Tech-stack
API | AWS Glue | AWS IAM | AWS KMS | AWS Lambda | Amazon S3 | Apache Spark | Azure DevOps | CI/CD | Databricks | Databricks CLI | Databricks Repos | Delta Lake | Docker | ELT | ETL | GitHub | Harness | Lakehouse | Power Apps | Power Automate | Power BI | PySpark | Python | SQL | Unity Catalog
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Applied AI Engineer - Bay Area USD 211K-263KArtificial Intelligence | C plus plus | C# | Embeddings | Feature Engineering401k | Comprehensive health and wellness benefits | Learning and development opportunities | Unlimited time offMid-level Full TimeHQ (San Francisco)24d ago
-
Principal Scientist, Machine Learning - Biomolecules USD 208K-286KAWS Batch | AWS ECS | AWS EKS | AWS S3 | AWS SageMakerAnnual incentive program | Healthcare coverage | Retirement benefitsSenior-level Full TimeCambridge, MA USA6h ago
-
Data Engineer Data Pipelines and ETL USD 99K-147KAnomaly Detection | Apache Airflow | CDC | Cloud Composer | Data Governance401k plan | Disability benefits | Life insurance | Life insurance coverage | Medical/Dental/VisionMid-level Full TimeBurbank, CA, US, 915057h ago
-
Sr. Software Engineer, Data Streaming Systems USD 130K-195KAutoscaling | Blocking I/O | CI/CD | Concurrency | Distributed Systems401k plan | Dental insurance | Disability benefits | Life insurance | Medical insuranceSenior-level Full TimeBurbank, CA, US, 915057h ago
-
Staff AI/ML Engineer USD 240K-270KAWS | Agentic Workflows | Azure | Data Curation | Deep learning401k | Commuter benefits | Dog-friendly office | Equity | FSA benefitsSenior-level Full TimeNew York City, NY9h ago
-
Staff AI/ML Engineer USD 240K-270KAWS | Agentic Workflows | Cloud platform | Deep learning | Foundation Models401k | Commuter benefits | Dog-friendly office | Equity | Flexible spending accountSenior-level Full TimeSan Francisco, CA9h ago
-
Senior AI/ML Engineer USD 240K-270KAWS | Agentic Workflows | Azure | Data Curation | Deep learning401k | Commuter benefits | Dog-friendly office | Equity | FSA benefitsSenior-level Full TimeNew York City, NY9h ago
-
Senior-level Full TimeUnited States10h ago
-
Embedded Device Firmware Engineer USD 145K-195KAutomated testing | C# | C++ | CI/CD | Circuit debuggingMid-level Full TimeSanta Clara, CA10h ago
-
Member of Technical Staff (Storage) USD 185K-200KAI Assisted Development | C++ | Concurrency Control | Data replication | Distributed SystemsDental insurance | Flexible time off | Life and disability insurance | Medical insurance | Mental wellbeing benefitsSenior-level Full TimeNew York, NY R10h ago
-
Senior Software Engineer - Data Platform USD 130K-220KAWS Lambda | AWS RDS | Airflow | Amundsen | Apache HiveHealth insurance | Parental leave | Professional development stipend | Remote workSenior-level Full TimeRemote - US R11h ago
-
Ai Engineer USD 100K-150KAI Agents | API Development | AWS | AWS Bedrock | Agentic Workflows401k | Commuter benefits | Dental insurance | Disability coverage | EAPMid-level Full TimeColumbia, MD, United States12h ago
-
Senior-level Contract Full TimeReston, VA, United States13h ago
-
Senior AI/ML Engineer: Python & Scientific Computing USD 175K-250KCelery | Distributed Computing | Flask | GraphQL | High PerformanceSenior-level Full TimeSan Francisco13h ago
-
Senior Robotics Software Engineer, Sensor Fusion USD 191K-253K*nix | C++ | CAN | Computer Vision | EthernetSenior-level Full TimeCosta Mesa, California, United States; Washington, …14h ago
-
Staff Robotics Software Engineer, Air Vehicle Autonomy USD 254K-338KAgent coordination | Computer Vision | Controls | Electrical Engineering | Embedded SystemsHealth benefits | Recovery BenefitsSenior-level Full TimeCosta Mesa, California, United States; Washington, …14h ago
-
Senior Data Engineer (Remote) USD 155KAgile | Apache Spark | BigQuery | Cassandra | Data Governance401k match | Dental insurance | Employee assistance program | Employee stock purchase plan | Flexible scheduleSenior-level Full TimeWork From Home, United States R15h ago
-
Senior AI Operations Engineer USD 170K-180KAI infrastructure | Azure | CI/CD | Cloud infrastructure | Container Engine for Kubernetes401k match | Employee assistance program | Employee stock purchase plan | Flexible schedule | Flexible spending accountSenior-level Full TimeWork From Home, United States R15h ago
-
Staff Software Engineer USD 190K-230KData Engineering | Data Pipelines | Database Design | Distributed Computing | ETLSenior-level Full TimeSan Francisco, California, United States16h ago
-
Data Engineer USD 89K-167KAmazon S3 | Apache Airflow | Apache Spark | DBT | Data ModelingDental insurance | Health insurance | Vision insuranceMid-level Full TimeRemote, USA R17h ago
-
Senior Data Engineer | Bankrate USD 100K-210KAPIs | AWS Lambda | AWS S3 | Airflow | Amazon EC2401k matching | Eastern Standard Time schedule | Employee assistance program | Flexible paid time off | Flexible spending accountsSenior-level Full TimeUnited States R17h ago
-
API Development | Airflow | Automated retraining | CI/CD | Cloud PlatformsEquityMid-level Full TimeNaples, United States17h ago
-
Adversarial Machine Learning | Anomaly Detection | Cloud Security | Machine Learning | PythonSecurity clearance premiumsMid-level Full TimeNaples, United States17h ago
-
AI Automation Engineer USD 96K-156KAPI Development | API Integration | Billing automation | Compliance Automation | ConnectwiseOnsite work authorization guidance required | Training and enablement support | Work with cross-functional teamsMid-level Full TimeGrand Rapids, Michigan18h ago
-
API Design | AWS | AWS Cloud | AWS Cloud Development Kit | AWS cloud developmentSenior-level ContractGlendale, United States18h ago