Lead Data Engineer
London ,London, United Kingdom
GBP 75K-95K (estimate) Senior-level Full Time
Tasks
- Architect data platforms
- Automate infrastructure with Infrastructure as Code
- Build data pipelines
- Build real-time streaming pipelines
- Conduct root cause analysis
- Develop Delta Lake pipelines
- Document architectures and processes
- Implement Change Data Feed
- Implement Spark Structured Streaming
- Implement metadata driven pipelines
- Manage Unity Catalog governance
- Mentor data engineers
- Optimize scalable analytics solutions
- Perform design/code reviews
- Set up CI/CD pipelines
- Translate business requirements into technical specs
- Troubleshoot data and code issues
- Tune SQL queries
- Use Databricks Asset Bundles
- Write T-SQL for data quality
Perks/Benefits
- N/A
Skills/Tech-stack
Asset bundles | Azure DevOps | Azure SQL | CI/CD | Change Data Feed | Data Governance | Data feed | Databricks | Databricks (asset) | Databricks Asset Bundles | Delta Lake | Delta Live Tables | Delta Live) | Git | Infrastructure as Code | PySpark | Python | SQL | Scala | Serverless SQL | Spark Structured Streaming | Structured Streaming | TSQL | Unity Catalog | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
AI Engineer GBP 60K-90KAWS | Azure | CI/CD | Databricks | DockerBike to work | Career progression pathways career development programme | Educational assistance | Employee assistance programme | Employee discountMid-level Full TimeLondon, Birmingham, Manchester, Newcastle upon Tyne, …17h ago
-
Contractor - Senior Data Engineer (3 months +) GBP 60K-80KAWS | AWS DMS | AWS Lambda | Amazon CloudWatch | Amazon KinesisSenior-level ContractLondon, United Kingdom21h ago
-
Batch Machine Learning | CI/CD | Causal Inference | Containerization | Data PipelinesEmployee discount | Employee sample sales | Flexible benefits allowance | Paid annual leave | Personalized learningSenior-level Full TimeLondon, England, United Kingdom22h ago
-
Data Engineer GBP 38K-42KAgile | Azure Data | Azure Data Factory | Azure Databricks | Azure Key VaultAnnual leave | Colleague mortgage | Cycle to work initiative | Discounted gym membership | Diverse and inclusive networksMid-level Full TimeSkipton - The Bailey, United Kingdom1d ago
-
MLOps Engineer GBP 76K-100KAirflow | Automated retraining | Autoscaling | BigQuery | Blue-Green DeploymentDiscounts | Enhanced parental leave | Holiday allowance | Learning and development programme | Life insuranceSenior-level Full TimeUnited Kingdom - Remote R1d ago
-
Mid-level Full TimeXC Birmingham Head Office, United Kingdom1d ago
-
AWS Step Functions | Agile | Amazon S3 | Amazon SQS | Apache AirflowLife assurance | Paid time off | Pension scheme | Training and developmentSenior-level Full TimeLondon - Central; Manchester; Cheltenham1d ago
-
Bioinformatician GBP 39K-44KArtificial Intelligence | CI/CD | Containerization | Data integration | Fair PrinciplesFlexible working arrangements | Generous paid time off | Onsite nursery | Pension scheme | Private medical insuranceMid-level Full TimeHinxton, United Kingdom1d ago
-
AI/ML Engineer, Responsible AI GBP 70K-80KBiomedical Data | Deep learning | Ethical AI | Foundation Models | Machine LearningSenior-level Full TimeLondon The Stanley Building, United Kingdom1d ago
-
Mid-level Full TimeGlasgow Campus, United Kingdom1d ago
-
Staff Applied Scientist GBP 165K-190KCloud Computing | Deep learning | Deep learning frameworks | Explainable AI | GitFlexible work schedule | Hybrid work | Mentoring & coaching | Share options | Team collaboration daysSenior-level Full TimeLondon2d ago
-
Senior Data Engineer GBP 81K-114KAWS EKS | Aerospike | Amazon Web Services | Apache Airflow | Apache FlinkDocumentation and onboarding support | Remote workSenior-level Full TimeLondon, England2d ago
-
AI architecture | APIs | Agent systems | Cloud platform | Data PipelinesSenior-level Full TimeLondon, UK2d ago
-
C++ | Data Processing | Data debugging | Data extraction | Distributed ComputingSenior-level Full TimeLondon, UK2d ago
-
Machine Learning Engineer GBP 110K-145KCI/CD | Docker | Feature Engineering | Kubernetes | Model Deployment401k matching | Dental insurance | Flexible paid time off | Health and wellness stipend | Health insuranceSenior-level Full TimeUnited Kingdom R3d ago
-
Airflow | BigQuery | Computer Vision | Deep learning | EmbeddingsBike to work scheme | Family planning support | Flexible vacation | Gender-affirming care | Income replacement programsSenior-level Full TimeRemote - United Kingdom R3d ago
-
Staff Machine Learning Engineer, ML Efficiency GBP 81K-118KApache Spark | C++ | Caching | Cloud Cost Optimization | Cloud infrastructureBike to work scheme | Family planning support | Flexible vacation | Gender-affirming care | Group pension with employer matchSenior-level Full TimeRemote - United Kingdom R3d ago
-
Senior-level Full TimeLondon3d ago
-
Lead Data Analyst - FinCrime Operations GBP 75K-115KData Modeling | Data Pipelines | Data Quality | Forecasting | KPI DevelopmentSenior-level Full TimeLondon, United Kingdom3d ago
-
Senior Data Engineer GBP 72K-88KApache Airflow | Async Processing | CI/CD | Cloud Build | Cloud ComposerCampus based work schedule | Global team collaboration | Hybrid working | Pathway to Lead Engineer | Support for certification and trainingSenior-level Full TimeDunton, Essex, United Kingdom3d ago
-
Senior Robotics Research Engineer GBP 62K-78KC# | C++ | CI/CD | Concurrent programming | Continuous DeploymentAnnual leave | Cycle to work scheme | EAP | Employer pension matching | Exclusive discountsSenior-level Full TimeWelwyn Garden City, United Kingdom R3d ago
-
Lead Data Quality Analyst GBP 62K-75KAlation | Collibra | Data Cataloging | Data Governance | Data QualityAnnual holiday allowance | Buy additional holiday | Colleague discount | Cycle to work scheme | Employee assistance programmeSenior-level Full TimeLondon, London, United Kingdom3d ago
-
Senior Robotics Software Engineer GBP 62K-84KAgile | Automated testing | BigQuery | CI/CD | CUDAAnnual leave | Cycle to work scheme | Employee assistance program | Exclusive discounts | Free shuttle busSenior-level Full TimeHatfield, United Kingdom R3d ago
-
Lead Data Scientist, Sports GBP 70K-75KCloud Computing | Machine Learning | Model Validation | Predictive Modeling | Probability theoryHybrid work policySenior-level Full TimeManchester, England, United Kingdom3d ago
-
Lead Data Scientist, Sports GBP 70K-75KCloud Computing | Data Modeling | Machine Learning | Probability theory | PyTorchHybrid working policySenior-level Full TimeStoke-on-Trent, England, United Kingdom3d ago