Principal Data Engineer - Databricks
Tasks
- Automate monitoring and reliability
- Collaborate on platform architecture
- Collaborate on security strategies
- Define job orchestration best practices
- Define workspace governance
- Design data pipelines
- Document development best practices
- Drive cost reduction
- Enforce fine grained access control
- Evaluate new Databricks features
- Implement CI/CD integration
- Implement Delta Live Tables
- Implement Unity Catalog governance
- Implement automated unit testing
- Implement data lineage
- Implement data pipelines
- Implement real-time streaming
- Mentor engineers
- Monitor Databricks costs
- Optimize Databricks performance
- Optimize Spark parameters
- Optimize cluster configuration
- Optimize job execution plans
- Standardize tagging and metadata
- Troubleshoot performance bottlenecks
- Troubleshoot streaming latency
Perks/Benefits
- 401k match program
- Annual bonus plans
- Free standby travel privileges
- Holiday pay programs
- Medical, dental, vision benefits
- Paid time off
Skills/Tech-stack
Apache Spark | Azure | Azure DevOps | CI/CD | Databricks | Delta Live Tables | Delta Live) | GitHub Actions | Infrastructure as Code | Kafka | Lakehouse Federation | MLflow | PySpark | Python | SQL | Structured Streaming | Terraform | Unity Catalog | “as-code”
Education
Roles
Data Engineer | Engineer | Principal | Principal Data Engineer
Related jobs
-
Big Data | Data Pipelines | Data cleaning | MAPE | Machine LearningMid-level Full TimeNewark, NJ, United States3h ago
-
Data Engineer II USD 84K-140KAgile | Amazon Redshift | Apache Airflow | CI/CD | Cloud ComputingMentorship | Professional developmentMid-level Full TimeBaltimore, Maryland, United States; McLean, Virginia, …4h ago
-
Databricks Senior Consultant USD 124K-207KBusiness Intelligence | Data Governance | Data Ingestion | Data Quality | Data TransformationMentorship | Professional development | TravelSenior-level Full TimeRosslyn, Virginia, United States; Washington, District …4h ago
-
Databricks Engineering Consultant USD 97K-162KApache Spark | Data Governance | Data Lineage | Databricks | ELTMentorship programs | Professional development | Travel opportunitiesMid-level Full TimeRosslyn, Virginia, United States; Washington, District …4h ago
-
Sr Lead Software Engineer - Java, Python, AI/ML USD 170K-189KAWS | Algorithms | Auto-GPT | Automated testing | Cause analysisSenior-level Full TimeJersey City, NJ, United States5h ago
-
Senior LLM Engineer USD 195K-255KAI Foundry | AI Search | Azure AI | Azure AI Foundry | Azure AI SearchSenior-level Full TimeAustin, Texas | Fremont, California | …5h ago
-
Mid-level Full TimeUS-Georgia-Augusta6h ago
-
Automation | C# | C++ | CPU Utilization | Capacity PlanningDevelopment workshops | Hands-on learning opportunities | Industry exposure | Mentorship | Social eventsEntry-level InternshipSan Jose, California, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Supply Chain & Logistics - LLM/Agent) - 2027 Start (PhD) USD 137K-246KAgents | Benchmarking | C++ | Deep learning | EvaluationEntry-level Full TimeSan Jose, California, United States6h ago
-
Machine Learning Engineer Graduate (E-Commerce Supply Chain & Logistics - LLM/Agent) - 2027 Start (PhD) USD 137K-246KAI Feedback | Agents | Benchmarking | C++ | DPOEntry-level Full TimeSeattle, Washington, United States6h ago
-
Agent evaluation | Classification | Clustering | Data Mining | Deep learningEntry-level Full TimeSeattle, Washington, United States6h ago
-
Apache Flink | Apache Hive | Apache Spark | Batch Processing | C++Entry-level Full TimeSan Jose, California, United States6h ago
-
Classification | Clustering | Data Mining | Deep learning | E-commerceEntry-level Full TimeSan Jose, California, United States6h ago
-
AI Workflow Orchestration | AI workflow | CD pipeline | CI Pipeline | CI/CDMid-level Full TimeSeattle, Washington, United States6h ago
-
Machine Learning Engineer (Technical Leadership) USD 213K-251KAB Testing | C plus plus | Data Mining | Data Pipelines | Deep learningSenior-level Full TimeMenlo Park, CA | New York, …7h ago
-
Software Developer, Scaled Ops AI Acceleration Team USD 170K-209KAgent Orchestration | Artificial Intelligence | Backend Development | Bug tracking | ComplianceSenior-level Full TimeAustin, TX | Menlo Park, CA …7h ago
-
Research Engineer, Machine Perception (Project Aria) USD 141K-208K3D Computer Vision | C++ | Computer Vision | Data Pipelines | Deep learningOnsite work | Work authorization supportMid-level Full TimeRedmond, WA7h ago
-
Software Engineer, Managed Service for Apache Spark USD 147K-210KApache Hadoop | Apache Spark | Apache YARN | BigQuery | Cloud platformMid-level Full TimeKirkland, WA, USA7h ago
-
Algorithms | Android | Android AppCompat | C# | C++Mid-level Full TimeWaterloo, ON, Canada; San Jose, CA, …7h ago
-
Staff Software Engineer, Applied AI, Model Quality USD 207K-300KData Processing | Data Storage | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeNew York, NY, USA7h ago
-
Software Engineer III, AI/ML GenAI, Search USD 147K-210KC++ | Data Processing | Debugging | Information Retrieval | LLMSenior-level Full TimeNew York, NY, USA7h ago
-
C++ | Cloud Computing | Data Processing | Debugging | Distributed ComputingSenior-level Full TimeMountain View, CA, USA7h ago
-
AI Inference | Benchmarking | C++ | Collective communication | Distributed SystemsSenior-level Full TimeMountain View, CA, USA7h ago
-
Customer Engineer III, Data Analytics, Google Cloud USD 152K-221KApache Spark | Architecture Design | Batch Processing | Big Data | Cloud NativeEquity compensation | Health insurance | Paid time off | Retirement benefitsSenior-level Full TimeNew York, NY, USA; Chicago, IL, …7h ago
-
Staff Software Engineer, Pixel Storage Architecture USD 207K-300KAOSP | Android | Block layer | DDR initialization | DRAMSenior-level Full TimeMountain View, CA, USA7h ago