Senior Databricks Migration Engineer
Tasks
- Apply Z Ordering data skipping liquid clustering and materialized views
- Architect dimensional models and star schemas for Power BI performance
- Build ETL ELT frameworks with PySpark
- Create Databricks Workflows for orchestration
- Create technical documentation and optimization playbooks
- Define governance standards for cluster sizing and auto scaling
- Deliver knowledge transfer for post migration self sufficiency
- Design bronze silver gold lakehouse architecture
- Enforce row level and column level security for Power BI
- Establish Delta Lake partition strategies and file size management
- Implement Delta Live Tables pipelines
- Implement cost monitoring dashboards
- Implement data security model with Unity Catalog
- Integrate Lakehouse security with Azure Active Directory and RBAC
- Lead technical data migration from legacy SQL to Databricks Lakehouse
- Optimize Databricks SQL Warehouses for high concurrency and low latency
- Provide technical leadership with code reviews workshops and pair programming
Perks/Benefits
Skills/Tech-stack
Active Directory | Apache Spark | Azure Active Directory | Data Quality | Data Skipping | Data loading | Data quality frameworks | Databricks | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | ELT | ETL | File Size Management | Incremental data loading | Liquid clustering | Materialized views | Metadata Driven | Metadata Driven Ingestion | Partitioning | Power BI | PySpark | Quality frameworks | RBAC | SQL Server | Star Schema | Unity Catalog | Z-Order
Education
Bachelor of Engineering | Bachelor of Science | Master of Science | PhD
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R14d ago
-
Associate AI Software Engineer USD 65K-100KAgile | Confluence | Data Analysis | Data Modeling | Data Visualization401k company match | Adoption Assistance | DailyPay | Dental insurance | Disability insuranceMid-level Full TimeSt. Louis Park, MN, United States5h ago
-
Mid-Level Data Engineer USD 90K-98KAPI Development | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageContract award contingent | Remote workMid-level Full TimeWork from home, VA, United States R6h ago
-
Senior-level Full TimeWork from home, VA, United States R6h ago
-
Data Engineer USD 145K-160KAWS | AWS Glue | Apache Airflow | Apache Spark | Data QualityContract award contingent | Hybrid work | Public trust clearance support | Technical documentation updates | Travel 15 percent or lessMid-level Full TimeFAIRFAX, VA, United States6h ago
-
Sr. Databricks Consultant USD 175K-250KAWS S3 | Apache Spark | Azure Data | Azure Data Lake | CI/CDRemote work | Secret clearance | Travel as neededSenior-level Full TimeWork from home, VA, United States R6h ago
-
Senior-level Full TimeWork from home, VA, United States R6h ago
-
Data Engineer USD 110K-165KCause analysis | Data Formats | Data Pipelines | Data Processing | Data TransformationCompetitive benefits | Hybrid remote onsite work | TS SCI clearance with poly eligibilitySenior-level Full TimeFAIRFAX, VA, United States6h ago
-
AI-Focused Red Team/Blue Team Operator USD 160K-205KAccess Management | Active Directory | Adversarial Machine Learning | Adversary Emulation | Alert triageSCI eligibility preferred | Top Secret clearanceMid-level Full TimeWashington, DC, United States6h ago
-
Data Engineer – Microsoft D365 F&O USD 119K-200KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Platform | Azure DevOpsMid-level Full TimeSaint Louis, MO, United States7h ago
-
Strategy & AI Consulting USD 129K-172KAWS | Artificial Intelligence | Azure | Cloud Computing | Cloud platformMid-level Full TimeAtlanta, United States8h ago
-
Data Engineer, gUP Engineering USD 106K-151KApache Flume | Apache Spark | Cloud Dataflow | Data Analysis | Data ModelingMid-level Full TimeBoulder, CO, USA11h ago
-
Data Engineer, GCS Data Science USD 106K-151KBig Data | C plus plus | Data Documentation | Data Modeling | Data ValidationEntry-level Full TimeNew York, NY, USA; Mountain View, …11h ago
-
GCP Data Engineering Lead USD 147K-190KAcxiom | Amperity | Apache Airflow | Apache Dataflow | BigQuery401k match | Dental insurance | Hybrid work arrangement | Medical insurance | Vision insuranceSenior-level Full TimeAtlanta, GA, United States16h ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | ClickHouse | Data Contracts | Data Modeling401k | Catered lunches | Commuter benefits | Continued education benefit | Equity compensationSenior-level Full TimeNew York, New York, United States17h ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | Data Contracts | Data Modeling | Data Quality401k | Catered lunches | Commuter benefits | Company wellness day off | Continued education benefitSenior-level Full TimeLos Angeles, California, United States17h ago
-
Senior Data Engineer USD 170K-200KApache Airflow | Apache Spark | Data Contracts | Data Modeling | Data Quality401k | Catered lunches | Commuter benefits | Continued education benefit | FSASenior-level Full TimeSan Francisco, California, United States17h ago
-
Azure Data Engineer USD 119K-219KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Autonomy | Career growth | Collaborative team | Community | Supportive cultureMid-level Full TimeIndianapolis, IN, US21h ago
-
Principal AI Engineer USD 140K-150KAI Search | Amazon Bedrock | Azure AI | Azure AI Search | Azure DataCommuter benefits | Flexible work arrangements | Insurance coverage | Learning and development training | Maternity leaveSenior-level Full TimeDallas, TX, United States22h ago
-
Data Engineer USD 90K-110KAlerting | Azure SQL | BI semantic models | CI/CD | Data ModelingHealth, dental, and vision insurance | Paid Holidays | Paid time off | Professional development supportMid-level Full TimeNorth Richland Hills, TX22h ago
-
Lead AI Engineer USD 140K-160KAirflow | Artificial Intelligence | CI/CD | Cloudera | Data orchestration401k plan | Dental insurance | Disability insurance | Health insurance | Life insuranceSenior-level Full TimeTexas, United States22h ago
-
Lead Analyst, Insights & Analytics - Collections USD 82K-172KAd-Hoc | Ad-hoc analysis | Dashboard Development | Data Integrity | Data TransformationSenior-level Full TimeMadisonville Office Building, United States22h ago
-
Lead, Software Engineer - Data Analyst TS/SCI USD 110K-218K.NET | AI Assisted Development | AI machine learning | AWS | AgileSenior-level Full TimePatrick Air Force Base, FL22h ago
-
Data Engineer USD 91K-169KData Modeling | Data Testing | Data Validation | ELT | ETL401k matching | Dental insurance | Global career opportunities | Health insurance | Paid time offMid-level Full TimeAustin, United States22h ago
-
Sr. Data Engineer - Remote USD 91K-163KAzure Data | Azure Data Factory | Batch Processing | CI/CD | Data Factory401k contribution | Comprehensive benefits package | Equity stock purchase | Flexible work schedule | Incentive and recognition programsSenior-level Full TimeLittle Rock, Arkansas, UNITED STATES; REMOTE R22h ago