Senior Databricks Migration Engineer
Tasks
- Build ETL/ELT frameworks
- Create dimensional models star schemas
- Create materialized views
- Create technical documentation and architecture diagrams
- Define cluster sizing and autoscaling governance
- Design bronze silver gold lakehouse architecture
- Develop Delta Live Tables pipelines
- Enforce row level security and column level security
- Implement Unity Catalog data governance
- Implement Z ordering data skipping liquid clustering
- Integrate Lakehouse security with Azure Active Directory and RBAC
- Lead code reviews workshops and pair programming
- Lead legacy to Databricks Lakehouse migration
- Manage Delta Lake partition strategy and file sizes
- Monitor Databricks DBU consumption
- Optimize Databricks SQL Warehouse for Power BI
- Perform data migration knowledge transfer
Perks/Benefits
Skills/Tech-stack
Active Directory | Azure Active Directory | Column-Level Security | Data Governance | Data Quality | Data Skipping | Data loading | Databricks | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | ELT | ETL | Entra ID | File Size Management | Incremental data loading | Liquid clustering | Materialized views | Microsoft Entra | Microsoft Entra ID | Partitioning | Power BI | PySpark | RBAC | Row Level Security | SQL Server | Star Schema | Unity Catalog | Z-Order
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R8d ago
-
Amazon S3 | Cloud Computing | Cybersecurity | Data Quality | Data integrationSenior-level ContractRaleigh, United States7h ago
-
Applied AI Engineer II - Encore Program USD 93K-170K.NET | AI Agent | AI agent orchestration | AWS | AWS BedrockMentorship | Professional growthMid-level Full TimeJersey City, New Jersey, United States; …7h ago
-
Analytics Engineer, Employee Technology Support USD 110K-130KCause analysis | Data Governance | Data Modeling | Data Monitoring | Data Quality401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceMid-level Full TimeNew York8h ago
-
Mid-level Full TimeIndianapolis, IN, 46254, USA10h ago
-
Software Development Staff Engineer Senior USD 184K-232KAWS | AWS CodeCommit | AWS CodeDeploy | AWS CodePipeline | AWS DocumentDBSenior-level Full TimeCosta Mesa, CA, United States18h ago
-
Database Developer Expert USD 183K-202KAlteryx | Amazon EMR | Amazon Redshift | Amazon S3 | Amazon Web ServicesSenior-level Full TimeCosta Mesa, CA, United States18h ago
-
Alerting | Cloud Computing | Compute Optimization | Data Quality | Distributed Training401k | Dental insurance | Health insurance | Office food and beverages | Stock option planMid-level Full TimeRWC HQ19h ago
-
Senior-level Full TimeUnited States - Remote R20h ago
-
Data Engineer USD 118K-142KAirflow | BigQuery | Cloud Composer | Cloud Dataflow | Cloud Dataproc401k retirement plan | Dental insurance | Educational assistance | Employee assistance program | Medical insuranceEntry-level Full TimeWHQ, United States20h ago
-
Senior-level Full TimeUnited States - Remote R20h ago
-
Machine Learning Data Engineer USD 100K-150KApache Beam | CI/CD | Data Drift | Data Lineage | Data ModelingRemote workSenior-level Full TimeUnited States - Remote R20h ago
-
APIs | Anomaly Detection | Data Visualization | Data acquisition | Databases401K and retirement plans | Adoption Assistance | Dental insurance | Disability coverage | Employee assistance programMid-level Full TimeShaw Plant 1G, 5225 Hwy 411 …20h ago
-
Machine Learning Research Engineer USD 100K-150KAccelerators | Computer Vision | Data Quality | Data quality monitoring | Dataset labelingCareer growth potential | Remote workSenior-level Full TimeUnited States - Remote R20h ago
-
Hadoop Solutions Developer USD 100K-150KAWS EMR | Airflow | Apache Atlas | Apache Flink | Apache HudiRemote workSenior-level Full TimeUnited States - Remote R20h ago
-
Developer, Compliance Data & Analytics USD 57K-85KAutomated reporting | Dashboards | Data Analysis | Data Modeling | Data Warehousing401k match | Backup child and elder care | EAP counseling sessions | Headspace access | Maternity & paternity leaveMid-level Full Time05555 - Casa del Mundo Office …20h ago
-
Data Engineer USD 61K-141KAWS | Apache Spark | Batch Processing | C++ | CloudFormationDependent care | Disability insurance | Health insurance | Life insurance | Paid leaveEntry-level Full TimeUSA, CA, San Diego (1615 Murray …20h ago
-
Artificial Intelligence Machine Learning Engineer 2 USD 130K-270KArtificial Intelligence | Data Modeling | Data Quality | Data Structures | Iterative design401k contribution | 401k vested from day 1 | AD&D insurance | Dental insurance | HSA contributionSenior-level Full TimeAnnapolis Junction, MD20h ago
-
Software Engineer 2 - Python/PySpark/AWS USD 130K-270KAWS Glue | AWS Lambda | Amazon DynamoDB | Amazon Web Services | PySpark401k contribution | AD and D insurance | Employee Dental Insurance | Employee Medical Insurance | Employee Vision InsuranceSenior-level Full TimeAnnapolis Junction, MD20h ago
-
API Integration | Apps Script | Automation | ChatGPT | Claude401k retirement plan | Dental insurance | Growth opportunities | Life insurance | Medical insuranceEntry-level Full Time InternshipGlendale, CA20h ago
-
Staff Data Warehouse Engineer USD 148K-209KApache Airflow | Cost Optimization | DBT | Data Governance | Data Lineage401k | Dental insurance | Disability insurance | Employee assistance program | Employee recognition programSenior-level Full TimeDraper, Utah, United States; San Jose, …21h ago
-
Lead Machine Learning Engineer USD 117K-236KAWS | Airflow | Artificial Intelligence | Batch Processing | CI/CD401k | Long-term incentive program | Medical/Dental/Vision insurance | Paid time offSenior-level Full TimeSan Diego, CA, United States21h ago
-
Machine Learning Engineer II USD 75K-85KAPI Development | AWS | Algorithms | Azure | Big Data401k match | Dental insurance | Disability insurance | Life insurance | Medical insuranceMid-level Full TimeSaint Charles, MO, United States21h ago
-
Data Engineer (Starlink Go-To-Market) USD 125K-175KAI Agents | APIs | Apollo | Artificial Intelligence | Billing Systems401k | Dental insurance | Employee stock purchase plan | Health insurance | Life insuranceMid-level Full TimeHawthorne, CA21h ago
-
Software Engineer, Platform USD 180K-225KAWS Step Functions | Amazon Web Services | Authentication and Authorization | CI/CD | CircleCICommuter stipend | Generous PTO | Health, dental and vision coverage | Learning and development stipend | Retirement benefitsMid-level Full TimeSan Francisco, CA; New York, NY22h ago