Lead Data Engineer
London ,London, United Kingdom
GBP 75K-95K (estimate) Senior-level Full Time
Tasks
- Architect data platforms
- Automate infrastructure with Infrastructure as Code
- Build data pipelines
- Build real-time streaming pipelines
- Conduct root cause analysis
- Develop Delta Lake pipelines
- Document architectures and processes
- Implement Change Data Feed
- Implement Spark Structured Streaming
- Implement metadata driven pipelines
- Manage Unity Catalog governance
- Mentor data engineers
- Optimize scalable analytics solutions
- Perform design/code reviews
- Set up CI/CD pipelines
- Translate business requirements into technical specs
- Troubleshoot data and code issues
- Tune SQL queries
- Use Databricks Asset Bundles
- Write T-SQL for data quality
Perks/Benefits
- N/A
Skills/Tech-stack
Asset bundles | Azure DevOps | Azure SQL | CI/CD | Change Data Feed | Data Governance | Data feed | Databricks | Databricks (asset) | Databricks Asset Bundles | Delta Lake | Delta Live Tables | Delta Live) | Git | Infrastructure as Code | PySpark | Python | SQL | Scala | Serverless SQL | Spark Structured Streaming | Structured Streaming | TSQL | Unity Catalog | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Benchmarking | C++ | CUDA | OpenCL | Operator fusionHybrid work | Work from homeSenior-level Full TimeLondon17h ago
-
AWS | Ansible | As-a-Service | Authentication | AutomationOn-call rotation support | RSUsSenior-level Full TimeLondon, United Kingdom18h ago
-
AWS | Ansible | Authentication | CDC | CI/CDMid-level Full TimeLondon, United Kingdom18h ago
-
Senior-level Full TimeCambridge, UK19h ago
-
Accuracy Checking | Alerting | Automation | Cause analysis | Completeness CheckingSenior-level Full TimeLondon19h ago
-
Team Leader - Document Research GBP 70K-71KAgreement Modeling | Calibration | Code review | Data Annotation | Data GovernanceSenior-level Full TimeLondon19h ago
-
Senior Data Engineer GBP 60K-70KAPI Integration | CI/CD | Cloud Computing | Data Quality | Data ValidationSenior-level Full TimeLondon19h ago
-
MLOps Engineer GBP 55K-55KCI/CD | Cloud Native | Data Governance | GitLab | GrafanaFlexible working arrangements | Hybrid workingMid-level Full TimeGB-ENG-LAN-Preston20h ago
-
Machine Learning Engineer GBP 26K-28KAlerting | Algorithms | Apache Spark | Azure | Cloud ComputingAdoption leave | Annual bonus scheme | Employee assistance programme | Employee wellbeing support | Free 247 virtual GP serviceMid-level Full TimeWelwyn Garden City, United Kingdom of …20h ago
-
Mid-level Full TimeLondon, England, GB22h ago
-
Senior Data Engineer - Data Science Platform GBP 60K-70KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageEmployee discount | Employee sample sales | Flexible benefits allowance | Paid annual leave | Personalized learningSenior-level Full TimeLondon, England, United Kingdom23h ago
-
Lead Data Scientist, Sports GBP 72K-80KCloud Computing | Computational Efficiency | Data Science | Machine Learning | Model ValidationHybrid workingSenior-level Full TimeStoke-on-Trent, England, United Kingdom23h ago
-
Staff MLOps Engineer (AI/ML Platform) GBP 70K-99KAWS EKS | Apache Spark | Data Drift | Databricks | Experiment trackingSenior-level Full TimeRemote, Remote, United Kingdom R1d ago
-
Power BI Developer GBP 51K-65KALM Toolkit | Active Directory | Azure Active Directory | DAX | DAX StudioFlexible working hours | Hybrid work | Work-life balanceSenior-level Full TimeGBR-Abingdon-Culham (057GB), United Kingdom1d ago
-
DevOps Engineer - ML GBP 58K-58KAI Services | AKS | Application Insights | Azure | Azure AI25 days holiday | Enhanced pension plan | Family-friendly benefits | Flexible working | Health insuranceMid-level Full TimeCentral London, United Kingdom1d ago
-
Mid-level Full TimeNorthampton, Barclays Campus, Pavilion Drive, United …1d ago
-
Analytics Engineer GBP 60K-70KAI tools | CI/CD | Code review | Containerization | DBT CoreAnnual leave | Barista bar | Company pension contributions | Cycle to work scheme | HealthcareMid-level Full TimeLondon, United Kingdom1d ago
-
Mid-level Full TimeCanary Wharf, 1 Churchill Place, United …1d ago
-
Senior Data Analytics Engineer GBP 70K-89KBusiness Intelligence | CI/CD | DBT | Data Documentation | Data ObservabilityCollaborative team | Documentation focused culture | Remote-friendly environmentSenior-level Full TimeRemote - United Kingdom R1d ago
-
Senior Machine Learning Engineer GBP 55K-55KAWS | Azure Machine Learning | Clustering | Databricks | Decision TreesHybrid working | Opportunities to lead coaching and mentoringSenior-level Full TimeManchester, Greater Manchester, United Kingdom1d ago
-
Legal Prompt Engineer GBP 50K-70KContract Law | Document Automation | LLM Testing | Language Models | Large Language ModelsMid-level Full TimeHybrid (SW1W 9SH, London, Greater London, … R1d ago
-
Senior AI Engineer GBP 72K-91KAKS | Azure AI | CI/CD | Containerization | GenAIOpsElectric car scheme | Employee discounts | Family leave | Flexible working options | Healthcare planSenior-level Full TimeLondon, England, United Kingdom1d ago
-
Senior AI Engineer GBP 72K-91KAKS | Azure AI | CI/CD | Containerization | GenAIOpsDiscounts | Electric car scheme | Family leave | Flexible working options | Generous pension contributionSenior-level Full TimeCardiff, Wales, United Kingdom1d ago
-
Senior AI Engineer GBP 72K-91KAI Services | AKS | Azure AI | Azure AI Services | CI/CDElectric car scheme | Employee discounts | Family leave | Flexible working | Healthcare planSenior-level Full TimeBrighton and Hove, England, United Kingdom1d ago
-
Senior-level Full TimeLondon, United Kingdom1d ago