Senior Databricks Platform Engineer
Tasks
- Apply data governance and metadata management
- Build and configure workspaces clusters notebooks jobs
- Build platform observability
- Collaborate with cloud infrastructure teams
- Configure metastore and catalog hierarchies
- Configure storage integrations
- Connect source systems to Databricks
- Develop user provisioning processes
- Enforce compute policies and cluster configurations
- Evaluate new AWS and Databricks features
- Implement RBAC and security groups
- Implement data lineage and auditing
- Implement encryption and data masking
- Implement secrets management
- Integrate AWS services with Databricks
- Integrate identity management with Databricks
- Maintain Databricks enterprise platform
- Maintain technical documentation
- Manage Unity Catalog governance
- Manage serverless SQL warehouses
- Monitor and manage workload reliability
- Monitor and tune cluster utilization
- Optimize costs and resource allocation
- Troubleshoot platform authentication and connectivity issues
- Validate security controls across data lifecycle
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Active Directory | Apache Hive | Apache Spark | Audit Logging | Autoscaling | Azure | Bash | Cluster configurations | Compute policies | Data Factory | Data Lineage | Data masking | Databricks | Delta Lake | EMR | Glue | Hadoop | IAM | KMS | KPI Dashboards | Monitoring | Powershell | Python | RBAC | S3 | SAML | SCIM | SQL | Scala | Secrets management | Serverless SQL | Unity Catalog | Workload management
Education
Related jobs
-
A/B | A/B Testing | API Integration | Airflow | Attribution ModelingCross-functional collaboration | Remote workSenior-level Full TimeUnited States R8h ago
-
Lead Data Engineer USD 125K-160KBigQuery | CI/CD | Cloud Composer | Cloud Storage | Cloud platform401k | Commuter benefits | Dental insurance | Disability coverage | Employee assistance programSenior-level Full TimeColumbia, MD, United States9h ago
-
Data Engineer - Databricks USD 172K-237KACID transactions | Airflow | Auto Loader | Azure Data | Azure Data FactoryMid-level Full TimeChicago, IL, United States10h ago
-
Senior Applied ML Engineer – LLMs & NLP USD 140K-175KConvolutional Neural Networks | Human-in-the-loop | Information Extraction | Java | Language ModelingDaily group classes | Free onsite fitness center | Free shuttle transportation | Gourmet cafeteriaSenior-level Full TimeEnglewood Cliffs, NEW JERSEY, United States10h ago
-
Senior AI Systems Engineer – LLM & Agentic Systems USD 140K-160KAWS | Agentic AI | Alerting | Drift Detection | EmbeddingsDaily group classes | Dry cleaning benefits | Free onsite fitness center | Free shuttle transportation | Gourmet cafeteriaSenior-level Full TimeEnglewood Cliffs, NEW JERSEY, United States R10h ago
-
Staff Data Scientist (AI/ML) USD 174K-278KAWS | Agile | Azure | BigQuery | Causal InferenceFlexible work optionsSenior-level Full TimeUS - Boston, MA ; US …10h ago
-
Lead ML Engineer USD 170K-210KAWS | Cloud infrastructure | Cloud platform | Computer Vision | Cost OptimizationExecutive-level Full TimeEnglewood Cliffs, NEW JERSEY, United States11h ago
-
Software Engineer 3 - Linux/Bash/Python/Apache Airflow/SQL/Jupyter Notebook/NumPy/JSON USD 130K-270KApache Airflow | Apache Spark | Bash | CSV | Confluence401k contribution | AD and D insurance | Dental insurance | Direct hire only | HSA contributionMid-level Full TimeAnnapolis Junction, MD12h ago
-
MapReduce | NoSQL | NumPy | Pandas | SQL401k contribution | Accidental death and dismemberment insurance | Health Savings Account contribution | Life insurance | Long-term disabilitySenior-level Full TimeAnnapolis Junction, MD13h ago
-
Ai Engineer USD 150K-170KApache Airflow | Artificial Intelligence | Azure Data | Azure Data Factory | Azure DevOps401k | Employee assistance program | Health spending account | Life insurance | Long-term disabilitySenior-level ContractBaltimore, Maryland, USA R13h ago
-
AI Search | Application Insights | Application diagnostics | Azure AI | Azure AI Search401k | Dental insurance | Educational assistance | Health savings account | Life insuranceMid-level Full TimeNew York, United States R14h ago
-
Intelligent Systems Engineer II USD 98K-166KAPIs | AWS Lambda | AWS S3 | Amazon Web Services | Artificial Intelligence401k match | Community service day | Dental insurance | Flexible spending accounts | Life insuranceMid-level Full TimeWest Chester, OH, United States15h ago
-
Intelligent Systems Engineer II USD 98K-166KAPI Gateway | APIs | AWS | Artificial Intelligence | Automated testing401k match | Community service day | Flexible spending accounts | Life insurance | Medical/Dental/Vision insuranceMid-level Full TimeWest Chester, OH, United States15h ago
-
Bash | Command Line | Command-line Interface | Data Tiering | Dell PowerScaleContract-to-hire | Hybrid work | Unlimited growthSenior-level Full TimeNashville, TN15h ago
-
Entry-level Full TimeUnited States16h ago
-
Senior AI Engineer USD 130K-200K.NET | Azure | Azure OpenAI | C# | Docker401k retirement plan | Employee assistance program | Health spending account | Life insurance | Long-term disabilitySenior-level ContractOmaha, Nebraska, USA R16h ago
-
Senior Backend Engineer USD 160K-220KAWS | Asynchronous processing | CI/CD | Cloud infrastructure | Debugging401k access | Career growth | Flexible PTO | Health, dental, vision coverage | Paid mealsSenior-level Full TimeSan Francisco, CA, US17h ago
-
Senior DevOps Engineer (Storage) - remote in the US USD 200K-300KAlerting | ArgoCD | CAPI | CSI | Cluster APICompany events | Conference attendance | Hackathons | Professional development and training | Tech talksExecutive-level Full TimeRemote, USA, United States R17h ago
-
Data Ops Engineer USD 130K-170KCI/CD | Incident Management | Log Management | Monitoring | ObservabilityOnsite work scheduleSenior-level Full TimeErie, PA, United States17h ago
-
AI Architect USD 111K-139KAI Search | Azure | Azure AI | Azure AI Search | Azure DevOps401k match | Employee discounts | Financial wellness planning | Great Place to Work certified | Health, dental and vision coverageSenior-level Full TimeCincinnati, Ohio, USA18h ago
-
AI Architect USD 148K-185KAI Search | Authentication | Azure | Azure AI | Azure AI Search401k match | Employee discounts | Financial wellness planning | Great Place to Work certification | Health, dental and vision coverageSenior-level Full TimeTampa, Florida, USA18h ago
-
Apache Flink | C++ | Data Lake | Data Warehouse | Data Warehouse ModelingSenior-level Full TimeSeattle, Washington, United States19h ago
-
Software Engineer III, AI/ML Machine Learning, Core USD 147K-210KAI/ML | AI/ML model development | Artificial Intelligence | C++ | CI/CDSenior-level Full TimeAustin, TX, USA20h ago
-
Imaging Machine Learning Engineer USD 132K-189K3A Tuning | Algorithm Development | Benchmarking | C++ | CalibrationMid-level Full TimeMountain View, CA, USA20h ago
-
Software Engineer III, AI/ML, AI and Infrastructure USD 147K-210KC++ | Code review | Data Processing | Data Storage | DebuggingSenior-level Full TimeSunnyvale, CA, USA20h ago