AWS Databricks Engineer
Tasks
- Build data ingestion and transformation pipelines
- Create data architecture documentation and blueprints
- Define CI CD and deployment monitoring architecture
- Define data models and source to target mappings
- Design Lakehouse architecture
- Design end-to-end data architecture
- Implement data governance security and access control
- Monitor data pipelines and troubleshoot failures
- Tune databricks clusters and optimize costs
Perks/Benefits
Skills/Tech-stack
ACID transactions | AWS Databricks | AWS Secrets | AWS Secrets Manager | AWS Step Functions | Access Control | Airflow | Amazon Aurora | Amazon CloudWatch | Amazon RDS | Amazon Redshift | Amazon Web Services | Apache Spark | Bronze Layer | CI/CD | Data Governance | Data Modeling | Data Quality | Databricks Jobs | Databricks Notebooks | Databricks Workflows | Delta Lake | DevOps | Gold Layer | Medallion Architecture | PySpark | Python | SQL | Schema evolution | Secrets Manager | Silver Layer | Spark SQL | Step Functions | Time Travel | Web Services
Education
N/A
Related jobs
-
Senior Software Engineer, Embedded UI GBP 70K-85KAPI Design | Artificial Intelligence | BrightScript | Cloud Development | Content ManagementCommuter benefits | Disability insurance | Financial wellness support | Healthcare benefits | Life insuranceSenior-level Full TimeCambridge, United Kingdom5h ago
-
AI systems | Agent systems | Agentic Workflows | Autonomous AI | Autonomous AI systemsSenior-level Full TimeUnited Kingdom, REMOTE, United Kingdom R17h ago
-
Lead Data Engineer GBP 70K-80KAI | AWS Redshift | Cloud Computing | DBT | Data ArchitectureCycle to work scheme | Electric vehicle scheme | Employee assistance programme | Employee fuel card scheme | Free breakfast dailySenior-level Full TimeCrewe, England, United Kingdom18h ago
-
Machine Learning Engineer GBP 82K-95KBatch Processing | CI/CD | Containerization | Distributed Systems | GPUsEmployee discount | Employee sample sales | Paid annual leave | Personalized learning | Private medical careSenior-level Full TimeLondon, England, United Kingdom19h ago
-
CI/CD | Containers | Data Pipelines | Data parallelism | Deep learningEmployee discount | Employee sample sales | Flexible benefits | Flexible benefits allowance | Paid annual leaveSenior-level Full TimeLondon, England, United Kingdom19h ago
-
Mid-level Full TimeGB-ENG-LAN-Preston21h ago
-
Lead Analytics Engineer GBP 97K-125KBigQuery | Data Governance | Data Modeling | Data Testing | Data WarehousingFlexible working hours | Learning budget | Relocation support | Visa sponsorshipSenior-level Full TimeCardiff, London or Remote (UK) R22h ago
-
Staff AI Engineer - AI Transformation GBP 90K-100KAPI Integration | Agent Orchestration | Agent systems | Backend Development | CachingSenior-level Full TimeLondon R22h ago
-
Senior Analytics Engineer GBP 76K-95KAgile | Airflow | BigQuery | Business Intelligence | DBTDistributed Working Within UK | Flexible working hours | Learning budget | Relocation support | Visa sponsorshipSenior-level Full TimeCardiff, London or Remote (UK) R22h ago
-
Production Engineer GBP 80K-105KAI workflows | Agent Orchestration | Bias Mitigation | C++ | Container ManagementDocumentation and capacity planning support | On-call rotationEntry-level Full TimeLondon, UK22h ago
-
Data Engineer - Kids Planet Central Support GBP 30K-33KAPI Development | AWS | Azure | Cloud platform | Data ComplianceAdoption leave | Anniversary Awards | Birthday leave | Career progression | Discounted childcareMid-level Full TimeLymm, England, WA13 0RN, GB22h ago
-
Senior AI Engineer - AI Transformation GBP 90K-105KAPI Integration | Batching | Caching | Chunking | Conversation memoryLife Event day | Paid time off | Private medical insurance | Volunteer days | Wellbeing daysSenior-level Full TimeLondon R22h ago
-
Research Engineer, Frontier Safety Mitigations, DeepMind GBP 225K-300KAI Safety | Adversarial Machine Learning | Anomaly Detection | Artificial Intelligence | Automated Red TeamingMid-level Full TimeLondon, UK22h ago
-
AI/ML Lead- Agentic AI GBP 81K-109KActive Learning | Clustering | Data Curation | Decision Trees | Deep learningSenior-level Full TimeGLASGOW, LANARKSHIRE, United Kingdom1d ago
-
Forward Deployed AI Engineer GBP 80K-109K.NET | APIs | Access Management | Agentic Workflows | Angular25 days annual leave | Critical illness protection | Cycle to work programme | Defined contribution pension with employer match | Dental and optical coverSenior-level Full TimeLondon, London, United Kingdom1d ago
-
Data Analyst (Analytics Engineer) GBP 69K-80KDBT | Data Governance | Data Lineage | Data Modeling | Data QualityElectric car scheme | Employee discounts | Family leave | Flexible working options | Healthcare planSenior-level Full TimeCardiff, Wales, United Kingdom1d ago
-
Software Engineering Specialist - Data GBP 70K-80KBatch Processing | Continuous integration | Data Ingestion | Data Modeling | Data PipelinesDental benefits | Gym membership | Healthcare benefits | Holiday purchase scheme | Paid carer’s leaveSenior-level Full TimeIpswich, GB, IP5 3RE1d ago
-
AI / Machine Learning Engineer GBP 50K-60KAPI Development | AWS | Azure | Cloud platform | Functional ProgrammingDental | Discounted mobile and broadband products | Gym memberships | Healthcare | Holiday purchase schemeMid-level Full TimeColchester, GB, CO2 8JY1d ago
-
Senior-level Full TimeLondon, United Kingdom1d ago
-
Data Engineer GBP 91K-106KAWS | Airflow | CI/CD | DBT | DagsterHybrid working | Office spaces | Remote work support | Supportive teamSenior-level Full TimeCheltenham, UK1d ago
-
Senior Applied Scientist GBP 100K-130KBlack | Cloud Computing | Cloud deployment | Complex Planning | Deep learningGenerous share options | Home office flexibility | Hybrid work schedule | Mentoring and coaching | Team daysSenior-level Full TimeLondon1d ago
-
Staff Applied Scientist GBP 165K-190KCloud deployment | Complex Planning | Data Analysis | Deep learning | Explainable AIFlexible office access | Generous share options | Hybrid work schedule | Mentoring and coaching | Team daysSenior-level Full TimeLondon1d ago
-
Lead Data Engineer GBP 70K-80KAWS | Agile | Apache Spark | Azure | CI/CDAnnual leave | Cycle to work | Employer pension | Family leave | Group life assuranceSenior-level Full TimeEdinburgh1d ago
-
Principal Data Engineer GBP 85K-110KAWS Kinesis | Amazon Redshift | Apache Airflow | Apache Flink | Apache HadoopSenior-level Full TimeLondon, United Kingdom1d ago
-
Mid-level Full TimeBOURNEMOUTH, DORSET, United Kingdom1d ago