Senior Databricks Migration Engineer
Tasks
- Align Lakehouse security architecture with Entra ID and RBAC
- Architect dimensional models and star schemas for Power BI performance
- Create technical documentation and optimization playbooks
- Define governance standards for clusters and serverless SQL compute
- Design ETL ELT frameworks with PySpark and Delta Live Tables
- Develop knowledge transfer framework post migration
- Enforce row level and column level security policies
- Establish Delta Lake partition and file size best practices
- Implement data optimization techniques including Z Ordering and data skipping
- Implement data security model with Unity Catalog
- Implement monitoring dashboards for Databricks Unit consumption
- Lead technical migration from legacy systems to Databricks Lakehouse
- Optimize Databricks SQL Warehouses for high concurrency queries
- Perform code reviews workshops and pair programming for Spark transition
- Translate relational warehousing paradigms to Lakehouse models
Perks/Benefits
Skills/Tech-stack
Active Directory | Apache Spark | Azure Active Directory | Cluster autoscaling | Data Quality | Data Skipping | Data loading | Databricks | Databricks Lakehouse | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | ELT | ETL | Entra ID | Incremental data loading | Liquid clustering | Materialized views | Microsoft Entra | Microsoft Entra ID | Power BI | PySpark | Python | RBAC | SQL | Serverless SQL | Star Schema | Unity Catalog | Z-ordering
Education
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R23d ago
-
Data Engineer USD 91K-114KANSI SQL | API | Amazon Redshift | Apigee | Azure401k with company match | Bereavement leave | Employee assistance program | Employee discount program | Family planning benefitsMid-level Full TimeRemote - Nationwide, United States R20h ago
-
Senior Data Engineer USD 111K-139KAPI Integration | Amazon Redshift | Apigee | Azure | BI Publisher401k match | Bereavement leave | Dental insurance | Employee assistance program | Employee discountsSenior-level Full TimeRemote - Nationwide, United States R20h ago
-
Data Platform Engineer-Solutions Delivery USD 125K-174KAPI Integration | API Management | APIs | Azure API | Azure API Management401k matching | Company holidays | Employee purchase program | Medical/Dental/Vision | Paid time offSenior-level Full TimeMerriam, KS, United States21h ago
-
Data Product Operations Lead USD 145K-175KData Pipelines | Data Quality | ETL | Metrics | Production Monitoring401k | Dental insurance | FSA | Gym classes | HSASenior-level Full TimeSan Francisco / New York City23h ago
-
Senior Machine Learning Research Engineer USD 210K-360KAPI Development | Audio Processing | Cloud Computing | Data Pipelines | Deep learning401k | Dental insurance | FSA | Gym classes | HSASenior-level Full TimeSan Francisco23h ago
-
Access Control | Azure | Azure Backup | Azure CLI | Azure DataHybrid work | Remote workSenior-level ContractSaint Paul, United States23h ago
-
Data Scientist Associate USD 77K-80KAPIs | Access Control | Artificial Intelligence | Audit Logging | BenchmarkingOnsite work locationMid-level Full TimeUniversity23h ago
-
Mid-level Full TimeUS-IL-Chicago1d ago
-
API | Airflow | Amazon Web Services | Apache Spark | AzureLearning and development | Mentorship | Travel opportunitiesEntry-level Full TimeAtlanta, Georgia, United States; Boston, Massachusetts, …1d ago
-
databricks engineer - Delivery Senior Consultant, Data Engineering and Conversion Solutions USD 124K-213KAPI | Agile | Data Governance | Data Modeling | Data QualityHybrid work model | Travel 10 percentSenior-level Full TimeLake Mary, Florida, United States1d ago
-
Mid-level Full TimeJersey City, New Jersey, United States1d ago
-
Cyber Data Scientist, Analytics and Insights USD 102K-171KCloud Computing | Data Modeling | Data Pipelines | Jira | Machine LearningMid-level Full TimeHuntsville, Alabama, United States1d ago
-
Research Engineer, Computer Vision USD 147K-208KAR/VR | Algorithm Development | Artificial Intelligence | C++ | Computer VisionSenior-level Full TimeSunnyvale, CA1d ago
-
Data Engineer, Product Analytics (University Grad) USD 173K-196KCompliance | Data Modeling | Data Quality | Data Security | Data TransformationCareer growth | Skill developmentNone Full TimeMenlo Park, CA | New York, …1d ago
-
Software Engineer, AI/ML, Google Search Ads USD 147K-210KAlgorithms | C++ | Code review | Data Processing | Data StorageMid-level Full TimeMountain View, CA, USA1d ago
-
Senior Staff Software Engineer, AI/ML, Google Ads USD 262K-364KC++ | Data Processing | Data Structures | Data structures algorithms | DebuggingSenior-level Full TimeMountain View, CA, USA1d ago
-
Applied Machine Learning | C++ | Concurrency | Distributed Systems | GoSenior-level Full TimeSan Jose, CA, USA1d ago
-
Customer Engineer III, Applied AI, Google Cloud USD 174K-252KAgent tooling | Application Performance Tuning | Application performance | C++ | CI/CDSenior-level Full TimeMountain View, CA, USA; New York, …1d ago
-
Senior Engineer, Big Data USD 134K-170KAmazon Web Services | Apache Spark | Automated testing | Azure | CI/CDSenior-level Full TimeUnited States1d ago
-
Software Engineer, Data Engineering USD 70K-200KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon EC2Mid-level Full TimeIrvine, CA1d ago
-
Senior-level Full TimeHouston, TX, US1d ago
-
AI Engineer II USD 60K-90KAPI Development | Backend Development | Cloud Computing | Databricks | DockerOnsite workMid-level Full TimeLocation Irving, Texas1d ago
-
APIs | AWS CloudWatch | AWS Kinesis | Agile | Amazon DynamoDB401k savings plan | Adoption benefits | Career development | Disability benefits | Employee assistance programSenior-level Full TimeIrving, Texas, United States1d ago
-
Visualization Engineer USD 128K-208KAnimation systems | C++ | GPU Optimization | Level of Detail | Lidar401k | Adoption benefits | Career development | Disability benefits | Flexible spending accountsSenior-level Full TimeIrving, Texas, United States1d ago