Data Engineer (Databricks), Assistant Vice President
Quincy, Massachusetts, United States
USD 110K-177K Executive-level Full Time
Tasks
- Analyze logs to resolve production issues
- Apply dimensional modeling and partitioning
- Automate Databricks asset deployment with Repos and CLI
- Automate processes with scripting
- Build and manage Databricks jobs workflows and notebooks
- Collaborate with IAM and security teams
- Collaborate with legal security compliance and data teams
- Containerize data workloads with Docker
- Continuously improve performance scalability and cost efficiency
- Design and maintain CI/CD pipelines
- Design and optimize Lakehouse architectures
- Design, build, and maintain scalable data pipelines
- Develop and optimize ETL ELT workflows on Databricks
- Develop reusable frameworks for data ingestion processing and orchestration
- Enable data exposure via APIs connectors and curated datasets
- Ensure adherence to data privacy and regulatory requirements
- Ensure data quality with unit testing and validation checks
- Implement data access controls and classification
- Implement data governance with Unity Catalog and AWS controls
- Implement lakehouse architecture with Bronze Silver Gold layers
- Integrate Databricks pipelines with Power Platform
- Integrate data from SQL Server and Oracle
- Maintain data lineage consistency and audit readiness
- Maintain documentation with architecture data flows and runbooks
- Monitor and troubleshoot distributed Spark workloads
- Monitor schedule and optimize Databricks workflows
- Process structured and semi-structured data
- Publish curated datasets for Power BI Power Apps and Power Automate
- Serve as Databricks and lakehouse architecture SME
- Translate business requirements into data engineering designs
- Tune Spark performance with caching and optimization
Perks/Benefits
- 401k match
- Dental insurance
- Employee assistance program
- Employee networks
- Flexible work/life support
- Health insurance
- Life insurance
- Long-term disability insurance
- Paid time off
- Paid volunteer days
- Retirement savings plan
- Vision insurance
Skills/Tech-stack
API | AWS Glue | AWS IAM | AWS KMS | AWS Lambda | Amazon S3 | Apache Spark | Azure DevOps | CI/CD | Databricks | Databricks CLI | Databricks Repos | Delta Lake | Docker | ELT | ETL | GitHub | Harness | Lakehouse | Power Apps | Power Automate | Power BI | PySpark | Python | SQL | Unity Catalog
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
AWS Cloud & Data Engineer – Healthcare Systems USD 90K-127KAPI Integration | AWS | Access Control | Audit Logging | AuroraDental insurance | Growth opportunities | Holidays | Medical insurance | Paid time offMid-level Full TimeRiverside, United States4h ago
-
API Integration | AWS ACM | Agile | Alerting | AnsibleCross-functional workshops | Hybrid work | Professional mentorship | Remote work flexibilitySenior-level ContractPittsburgh, United States R4h ago
-
Senior-level Full TimeUS-TX-Irving4h ago
-
Low Power Design Methodology and Optimization Engineer USD 163K-237KCPF | CPU Power Optimization | Logic synthesis | Low power | Low power designSenior-level Full TimeAustin, TX, USA5h ago
-
Software Engineer, Managed Service for Apache Spark USD 147K-211KAPI Integration | Apache Flink | Apache Hadoop | Apache Spark | Apache YARNMid-level Full TimeKirkland, WA, USA5h ago
-
Staff Software Engineer, Cooling Optimization USD 207K-300KC++ | Compute Technologies | Control Theory | Cooling systems | Data StructuresSenior-level Full TimeSunnyvale, CA, USA5h ago
-
Senior Software Engineer, AI/ML, Creative Intelligence USD 174K-252KAlgorithms | C++ | Data Processing | Data Structures | Deep learningSenior-level Full TimeMountain View, CA, USA5h ago
-
Data Engineer, Product Data Warehouse, Go-To-Market USD 156K-226KApache Flume | Apache Spark | Business Intelligence | Code review | DashboardsMid-level Full TimeNew York, NY, USA; Atlanta, GA, …5h ago
-
Mid-level Full TimeMountain View, CA, USA6h ago
-
AI Engineer - Healthcare USD 125K-184KAI Services | APIs | Auditability | Azure AI | Azure AI ServicesFlexible work location | Paid maternity & parental leave | Paid sick leave | Paid vacation | Remote workMid-level Full TimeNS, CA8h ago
-
Genome Editing Pipeline Data Scientist USD 94K-141KAI Model Deployment | AI model | Analytics | Bias Mitigation | Business IntelligenceDental insurance | Health insurance | Paid time off | Retirement plan | Sick leaveMid-level Full TimeChesterfield, Missouri, US8h ago
-
Senior AI Engineer USD 139K-229KAnt | Apache Lucene | Apache Solr | Big Data | Configuration ManagementHealth and wellness programs | Time offSenior-level Full TimeSunnyvale, CA, United States14h ago
-
Senior Software Engineer/Computer Scientist USD 145K-170KC# | C++ | Configuration Management | Continuous integration | Distributed SystemsEmployee-owned company | Onsite work | Reasonable accommodationSenior-level Full TimeOrlando, FL, US15h ago
-
Staff Machine Learning Engineer 2, Ads USD 159K-309KAWS | Airflow | Apache Spark | BigQuery | Cloud Platforms401k plan company match | Disability insurance | Electric Car Charging Station | Employee assistance program | Flexible spending accountSenior-level Full TimeMountain View, USA16h ago
-
Staff Machine Learning Engineer 2, Ads USD 164K-282KAWS | Airflow | Amazon SageMaker | Apache Spark | BigQuery401k plan with company match | Dental insurance | Disability insurance | Electric car charging | Employee assistance programSenior-level Full TimeMountain View, USA16h ago
-
Associate Director, Biostatistics & AI USD 173K-217K21 CFR | 21 CFR Part 11 | ADaM | Adaptive Design | Annex 11401k employer match | Company provided life and disability | Comprehensive health care | Employee stock purchase program | Flex Spending AccountsMid-level Full TimeRemote - USA R16h ago
-
Senior Computational Fluid Dynamics Engineer USD 100K-190KANSYS-FLUENT | Computational Fluid Dynamics | Data Preprocessing | Data postprocessing | Fluid Dynamics401k | Bonuses | Equity | FSA | Flexible time offSenior-level Full TimeSanta Clara, CA or Remote R17h ago
-
Data Analysis | Deep learning | GenAI | Langchain | Language ModelsFreelance project-based work | Part-time hours | Project-based compensationMid-level FreelanceUnited States - Remote R17h ago
-
Freelance Machine Learning Engineer USD 180KLLMs | Langchain | MLOps | Machine Learning | NumPyFreelance engagement | Part-time project-based workMid-level FreelanceUnited States - Remote R17h ago
-
Database operations | LLMs | Langchain | MLOps | Machine LearningPaid per project | Part-time flexible schedule | Project based workMid-level FreelanceNew York, United States - Remote R17h ago
-
Langchain | Language Models | Large Language Models | MLOps | NumPyEnglish proficiency support | Flexible workload during active phases | Part-time schedule | Project based workMid-level FreelanceTexas, United States - Remote R17h ago
-
Freelance Machine Learning Engineer USD 180KLangchain | Language Models | Large Language Models | MLOps | Machine LearningFlexible weekly hours during active phases | Part-time availability | Project based workMid-level FreelanceNew York, United States - Remote R17h ago
-
Freelance Machine Learning Engineer USD 180KLangchain | MLOps | Machine Learning | NumPy | PandasProject based workMid-level FreelanceTexas, United States - Remote R17h ago
-
Senior-level Full TimeSan Jose, United States R17h ago
-
Sr Embedded Software Test Engineer - Abiomed USD 105K-169KBehavior-Driven Development | CI/CD | Case design | Cause analysis | Embedded SystemsSenior-level Full TimeUS345 MA Danvers - 22 Cherry …17h ago