Lead Data Engineer
London ,London, United Kingdom
GBP 75K-95K (estimate) Senior-level Full Time
Tasks
- Architect data platforms
- Automate infrastructure with Infrastructure as Code
- Build data pipelines
- Build real-time streaming pipelines
- Conduct root cause analysis
- Develop Delta Lake pipelines
- Document architectures and processes
- Implement Change Data Feed
- Implement Spark Structured Streaming
- Implement metadata driven pipelines
- Manage Unity Catalog governance
- Mentor data engineers
- Optimize scalable analytics solutions
- Perform design/code reviews
- Set up CI/CD pipelines
- Translate business requirements into technical specs
- Troubleshoot data and code issues
- Tune SQL queries
- Use Databricks Asset Bundles
- Write T-SQL for data quality
Perks/Benefits
- N/A
Skills/Tech-stack
Asset bundles | Azure DevOps | Azure SQL | CI/CD | Change Data Feed | Data Governance | Data feed | Databricks | Databricks (asset) | Databricks Asset Bundles | Delta Lake | Delta Live Tables | Delta Live) | Git | Infrastructure as Code | PySpark | Python | SQL | Scala | Serverless SQL | Spark Structured Streaming | Structured Streaming | TSQL | Unity Catalog | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Career growth opportunity | Friendly work environment | Impact on consumer and enterprise products | Remote friendly 100% distributed settingMid-level Full TimeLondon, United Kingdom19h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Entrepreneurial environment | Flexible management structure | Friendly laid-back atmosphereMid-level Full TimeBrighton, United Kingdom19h ago
-
Internal Audit AVP - Data Analytics GBP 47K-58KAWS | Alteryx | Audit Testing | Cloud Computing | DashboardsEntry-level Full TimeCanary Wharf, 1 Churchill Place, United …1d ago
-
AI/ML/Data Science Engineer GBP 50K-60KAnomaly Detection | Cloud Platforms | Computer Vision | Computer Vision Video Analytics | Data PreprocessingAnnual leave | Bank Holiday Leave | Dental care | Discounts | Enhanced maternity/paternity leaveMid-level Full TimeBiggin Hill, United Kingdom1d ago
-
LLM Specialist / AI Developer GBP 59K-65KAPI Security | Azure OpenAI | Copilot Studio | Evaluation | GroundingFixed term position | Hybrid working | Working permit sponsorshipMid-level Full TimeLondon, Greater London, United Kingdom1d ago
-
Senior GenAI Software Engineer GBP 76K-100KA/B | A/B Testing | B testing | Debugging | Diffusion ModelsDental insurance | Equity | Health insurance | Vision insurance | Wellness stipendsSenior-level Full TimeLondon, UK R1d ago
-
Senior-level Full TimeUnited Kingdom (Remote) R1d ago
-
Agile | Cloud Platforms | Data Architecture | Data Observability | Data QualityHybrid workingSenior-level Full TimeUnited Kingdom, United Kingdom1d ago
-
Data Science Lead - Logistics GBP 62K-80KArtificial Intelligence | Big Data | Cloud | IoT | JavaAnnual leave | Buy As You Earn Scheme | Cycle to work scheme | Employee assistance programme | Employee discountsSenior-level Full TimeLondon, United Kingdom R1d ago
-
Senior Data Engineer BI Hub GBP 62K-77KAWS | Agile | Alerting | Azure | CI/CDAnnual holiday allowance | Buy additional holiday | Colleague discount | Cycle to work scheme | DiscountsSenior-level Full TimeLondon, London, United Kingdom1d ago
-
AWS | Amazon Bedrock | Azure | Azure OpenAI | DockerMid-level Full TimeLONDON, LONDON, United Kingdom1d ago
-
Applied AI ML Lead - LLM Suite Engineering GBP 72K-95KAPIs | AWS | AWS Bedrock | Agent communication | Agent to AgentSenior-level Full TimeLONDON, LONDON, United Kingdom1d ago
-
Senior AI Engineer GBP 75K-75KAWS | Agent systems | Artificial Intelligence | Azure | CI/CDAnnual bonus | Discounted gym membership | Electric vehicle leasing | Experience days | Hybrid workSenior-level Full TimeLondon, United Kingdom R1d ago
-
Full Stack AI Engineer GBP 78K-80KAlembic | Application Insights | Asynchronous programming | Auditability | AuthenticationMid-level Full TimeBelfast1d ago
-
Principal Computer Vision Engineer GBP 67K-109K3D Object Tracking | ARM | C# | C++ | Camera ModelsHybrid work | Mentoring | Professional developmentSenior-level Full TimeLondon, UK1d ago
-
Senior Data Engineer GBP 77K-80KAWS | AWS Glue | AWS Lambda | AWS S3 | Access ControlAnnual bonus programme | Annual learning stipend | Birthday day off | Collaborative Agile culture | Flexible working hoursSenior-level Full TimeLondon, United Kingdom1d ago
-
System Engineer - PostgreSQL GBP 150KDebian | Linux | MySQL | PostgreSQL | PythonBonus pay | Hybrid workMid-level Full TimeLondon, England, United Kingdom1d ago
-
Senior HPC Storage Engineer GBP 65K-80KCephFS | GPFS | Linux | Lustre | NASHybrid working | Quarterly bonuses | Signing bonusSenior-level Full TimeLondon, England, United Kingdom1d ago
-
Senior HPC Software Engineer GBP 65K-80KC++ | Cloud Storage | Databases | Deployment Monitoring | Distributed SystemsFlexible budgetSenior-level Full TimeLondon, England, United Kingdom1d ago
-
Data Mart | Data Modeling | Data Warehousing | HDFS | Hadoop100 percent remote | Outside IR35Mid-level Full TimeCoventry, England, United Kingdom R1d ago
-
Data Engineer GBP 60K-70KAPI | AWS | ETL | Python | SnowflakeCareer development | Health insurance | Life insurance | Paid time off | Paid volunteering leaveMid-level Full TimeManchester, England, United Kingdom1d ago
-
AI Engineer / Machine Learning Engineer GBP 85K-100KAlgorithms | Cloud Computing | Data Science | Data Structures | DockerSenior-level Full TimeLondon, England, United Kingdom1d ago
-
Data Engineer (GCP) GBP 70K-80KCI/CD | Cloud Composer | Cloud Functions | Cloud Storage | Cloud platformMid-level Full TimeManchester, England, United Kingdom1d ago
-
Data Engineer GCP (Remote) GBP 70K-80KAlerting | CI/CD | Cloud Composer | Cloud Functions | Cloud StorageRemote workMid-level Full TimeStoke-on-Trent, England, United Kingdom R1d ago
-
Entry-level Full TimeStoke-on-Trent, England, United Kingdom1d ago