Senior Databricks Migration Engineer
Tasks
- Apply Z Ordering data skipping liquid clustering and materialized views
- Architect dimensional models and star schemas for Power BI performance
- Build ETL ELT frameworks with PySpark
- Create Databricks Workflows for orchestration
- Create technical documentation and optimization playbooks
- Define governance standards for cluster sizing and auto scaling
- Deliver knowledge transfer for post migration self sufficiency
- Design bronze silver gold lakehouse architecture
- Enforce row level and column level security for Power BI
- Establish Delta Lake partition strategies and file size management
- Implement Delta Live Tables pipelines
- Implement cost monitoring dashboards
- Implement data security model with Unity Catalog
- Integrate Lakehouse security with Azure Active Directory and RBAC
- Lead technical data migration from legacy SQL to Databricks Lakehouse
- Optimize Databricks SQL Warehouses for high concurrency and low latency
- Provide technical leadership with code reviews workshops and pair programming
Perks/Benefits
Skills/Tech-stack
Active Directory | Apache Spark | Azure Active Directory | Data Quality | Data Skipping | Data loading | Data quality frameworks | Databricks | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | ELT | ETL | File Size Management | Incremental data loading | Liquid clustering | Materialized views | Metadata Driven | Metadata Driven Ingestion | Partitioning | Power BI | PySpark | Quality frameworks | RBAC | SQL Server | Star Schema | Unity Catalog | Z-Order
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Related jobs
-
Senior-level Full TimeRedwood City, California, United States12h ago
-
Data Engineer I - Salesforce Data 360 USD 87K-123KAPIs | Access Controls | Data Governance | Data Lineage | Data Modeling401k match | Career growth and development programs | Comprehensive onboarding | Conference participation | Employee resource groupsMid-level Full TimeMorrisville, North Carolina (NC), United States R13h ago
-
AWS | AWS CDK | AWS CloudFormation | AWS Database | AWS Database Migration ServiceSenior-level Full TimeSt. Louis, MO14h ago
-
AWS | AWS CDK | AWS DMS | AWS Glue | AWS LambdaSenior-level Full TimeSt. Louis, MO14h ago
-
Senior Data Engineer USD 149K-180KAPI Architecture | Apache Airflow | Cloud Architecture | Data Ingestion | Data Ingestion PipelinesCollaborative team | Four days per week office schedule | Mission-driven workSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL15h ago
-
Senior Data Engineer USD 150K-170KAlerting | Data Processing | Data Quality | Data Warehousing | Data orchestrationSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL17h ago
-
Database Administrator - Data Engineer USD 90K-110KAccess Controls | Data Reconciliation | Data Validation | Data integration | Database DesignCareer guidance | Mentoring | Training opportunitiesMid-level Full TimeBrooklyn, NY, US17h ago
-
Angular | Autogen | CI/CD | Databricks | DockerMentoring system | Professional developmentSenior-level Full TimeJersey City, NJ, United States17h ago
-
Data Engineer II USD 92K-176KAWS | Apache Airflow | Azure | Azure Data | Azure Data Factory401k matching | Dental insurance | Health insurance | Onsite fitness center | Time offSenior-level Full TimeUS-Rhode Island-Providence17h ago
-
A/B | A/B Testing | Apache Flink | Apache Spark | B testingEntry-level Full TimeSan Jose, California, United States18h ago
-
Apache Flink | Apache Spark | C plus plus | Data Mining | Data ModelingEntry-level Full TimeSan Jose, California, United States18h ago
-
Software Engineer, Cloud Bigtable SQL and Analytics USD 147K-210KApache Beam | Apache Spark | C++ | Concurrency | Consistency modelsBenefits package | On-call rotationMid-level Full TimeNew York, NY, USA19h ago
-
Senior-level Full TimeMountain View, CA1d ago
-
Databricks Platform Engineer USD 115K-135KAWS | Asset bundles | Audit Logs | Azure Databricks | CI/CD401k match | Employee assistance resources | Medical/Dental/Vision insurance | Paid time off | Professional development supportMid-level Full TimeRemote, United States R1d ago
-
Analytics Engineer USD 140K-190KCI/CD | DBT | Data Modeling | Data Quality | Data Testing401k match | Flexible time off | Growth opportunities | Health benefits | Remote-friendlySenior-level Full TimeUnited States (Remote) R1d ago
-
Early Career Consult Program - Associate AI Engineer USD 129K-279KAWS | Azure | CI/CD | Cloud Computing | Data AnalysisAccess to learning opportunities | Be Well programs | Coaching | Continuous feedback | Employee referral programMid-level Full TimeUSDALFRI - Dallas Frisco AI HUB … R1d ago
-
AI | AWS | Automation | Azure | Data AnalysisOnsite 3 Days Per Week Requirement | Remote workMid-level Full TimeSouth Carolina Work at Home, United … R1d ago
-
Data Engineering Tech Lead USD 10KAPI Development | Apache Spark | Automated unit testing | Azure | Azure DataSenior-level Full TimeWORK AT HOME - OHIO, United … R1d ago
-
Senior Data Engineer USD 106K-176KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage401k retirement plan | Employee assistance program | Flexible spending accounts | Health savings account | Long-term disabilitySenior-level Full TimeGH Office: Tysons Corner, VA (Headquarters), …1d ago
-
Data Engineer - Mid USD 106K-176KABAC | API Gateway | AWS Glue | AWS IAM | AWS S3401k retirement plan | Employee assistance program | Employee referral program | Flexible benefits package | Medical/Dental/Vision insuranceSenior-level Full TimeGH Office: Tysons Corner, VA (Headquarters), …1d ago
-
AI Engineer II USD 60K-90KAPIs | Backend Development | Databricks | Deep learning | DockerOnsite workMid-level Full TimeLocation Irving, Texas1d ago
-
Senior Cloud Data Engineer - Remote USD 91K-163KAgile | Airflow | Apache Spark | Azure | Azure Data4 days per week office requirement in specific locations | Development opportunities | Flexible remote work | Performance recognition | Remote work within U.S.Senior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R1d ago
-
Senior Data Engineer USD 91K-163KAPIs | Access Control | Batch Processing | Cloud Data | Cloud Data Platforms401k contribution | Career development opportunities | Comprehensive benefits package | Equity stock purchase | Hybrid work scheduleSenior-level Full TimePrimary location: Eden Prairie, Minnesota1d ago
-
Data Engineer (Hybrid) USD 86K-165KAWS | AWS Glue | Access Control | Agile | Amazon EMR401k matching | Adoption Assistance | Child and elder care services | Disability coverage | Employee assistance planMid-level Full TimeUS-IA-CEDAR RAPIDS-193 ~ 1120 Collins Rd … R1d ago
-
Analyst - Data Engineering USD 95KAPI Integration | AWS | Apache Spark | Automated testing | AzureHybrid work schedule | Inclusive and diverse work culture | Professional development opportunities | Supportive leadershipMid-level Full TimeLouisville - West Market, United States R1d ago