Lead Data Engineer
GBP 64K-79K (estimate) Senior-level Full Time
Tasks
- Build Delta Live Tables using PySpark
- Build data platform on Azure
- Create CI CD pipelines for data engineering
- Design data pipelines
- Develop ETL/ELT processes
- Develop enterprise data model
- Ensure high availability
- Implement data quality and governance
- Implement streaming data solutions
- Integrate third-party APIs and data sources
- Mentor junior engineers
- Monitor data platform
- Optimize pipeline performance and reliability
- Troubleshoot complex data engineering issues
Perks/Benefits
- N/A
Skills/Tech-stack
API Integration | Azure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Databricks | Azure DevOps | CI/CD | Cosmos DB | Data Factory | Data Governance | Data Lake Storage | Data Modeling | Data Quality | Data Security | Delta Lake | Delta Live Tables | Delta Live) | ELT | ETL | Git | Infrastructure as Code | Monitoring | NoSQL | PySpark | Python | Relational databases | SQL | Streaming | “as-code”
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
APIs | Agent systems | Cloud platform | Cost Per | Cost Per RequestSenior-level Full TimeLondon, UK5h ago
-
Amazon SageMaker | Autoscaling | Capacity Planning | Cause analysis | Cloud infrastructureSenior-level Full TimeGLASGOW, LANARKSHIRE, United Kingdom6h ago
-
Staff Software Engineer - Experimentation GBP 24K-26KAPI Design | API Governance | Automation | Azure | Distributed SystemsAdoption leave | Colleague discount | Exclusive perks and savings | Family policies | Holiday allowanceSenior-level Full TimeUnited Kingdom7h ago
-
Embedded Software Engineer GBP 57K-63KC# | C++ | CAN | Cause analysis | DebuggingDisability programs | Life insurance | Private medical care | Remote Work Hybrid Schedule | Work-life balance programsMid-level Full TimeGB-Ashby de la Zouch-Power House, Excelsior …16h ago
-
AI DevOps Engineer GBP 84K-95KAPI Gateway | AWS Bedrock | AWS ECS | AWS EKS | AWS LambdaCareer development opportunities | Employee recognition programs | Health and wellness benefits | Inclusive culture | Training and education supportMid-level Full TimeLondon, United Kingdom16h ago
-
Founding AI Engineer (Staff) GBP 65K-120KAWS | ClickHouse | Distributed Systems | Incident Management | Langgraph30 days annual leave | Employee Health Insurance | Regular team offsites and trips | Visa sponsorship | Weekly team dinners and socialsSenior-level Full TimeLondon1d ago
-
Staff Applied Scientist GBP 110K-160KCloud Computing | Data Analysis | Deep learning | Git | GitHubFlexible remote work | Hybrid work model | Mentoring and coaching | Share options | Team meetups via Slack and ZoomSenior-level Full TimeLondon1d ago
-
Data Engineer GBP 52K-70KAWS | Apache Flink | Apache Spark | Cloud platform | ElasticsearchHealth and wellness benefits | Opportunity for career growth | Remote work option | Work with cutting-edge technologyMid-level Full TimeLondon R1d ago
-
AI Engineer USD 100K-250KBackend Development | Context Management | Evaluation Pipeline | Postgres | Prompt engineeringEquity | Free meals | Private medical insurance | Visa sponsorshipEntry-level Full TimeLondon2d ago
-
AWS CDK | AWS Glue | AWS Lambda | Agile | Amazon KinesisGlobal experience | Immigration supportMid-level Full TimeLondon, United Kingdom2d ago
-
C++ | Data Mining | Distributed Computing | Generative AI | High PerformanceGlobal experience | Immigration supportMid-level Full TimeLondon, United Kingdom2d ago
-
Research Engineer, Safety Oversight, DeepMind GBP 100K-180KAutomated Evaluation | Classifier Models | Data Pipelines | Data Quality | ExperimentationMid-level Full TimeLondon, UK2d ago
-
Data Engineer GBP 46K-52KAzure | Azure Databricks | CI/CD | PythonBonus scheme | Discounts and perks | Employee assistance programme | Generous holiday entitlement | Hybrid workingMid-level Full TimeSentinel Building, Glasgow, United Kingdom2d ago
-
Sr. Specialist Solutions Architect GBP 78K-120KAI Observability | AWS | Agent Orchestration | Agentic Systems | Apache SparkSenior-level Full TimeLondon, United Kingdom2d ago
-
Lead Data Engineer GBP 70K-75KAWS DMS | Amazon Web Services | BigQuery | CDC | CI/CDHybrid work environmentSenior-level Full TimeLondon, UK R2d ago
-
Data Analyst, Product GBP 95K-124KA/B | A/B Testing | Analytical Functions | Apache Airflow | B testingSenior-level Full TimeUnited Kingdom2d ago
-
Lead Product Data Analyst GBP 60K-60KA/B | A/B Testing | B testing | Excel | ExperimentationAnnual leave | Corporate gym membership | Cycle to work scheme | Dental insurance | Eye TestsSenior-level Full TimeLondon, United Kingdom3d ago
-
Machine Learning Engineer GBP 55K-55KAzure Machine Learning | Clustering | Data Monitoring | Decision Trees | DevOpsHybrid working | Opportunities for coaching and mentoringMid-level Full TimeManchester, Greater Manchester, United Kingdom3d ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Flexible management | Remote workMid-level Full TimeLondon, United Kingdom3d ago
-
Data Business Engineer GBP 55K-65KAmazon Redshift | BigQuery | Data Modeling | Data Validation | Data WarehousingMid-level Full TimeLondon, United Kingdom3d ago
-
Senior Data Systems Software Engineer GBP 80K-106KA/B | A/B Testing | API | API Versioning | Application ExpressFlexible medical insurance | Life insurance | Retirement options | Volunteer programsSenior-level Full TimeLONDON, United Kingdom3d ago
-
Blameless postmortems | C++ | Distributed Systems | Fault Tolerance | JavaSenior-level Full TimeLondon3d ago
-
Applied statistics | Artificial Intelligence | Digital Signal | Digital Signal Processing | Edge ComputingHybrid workingSenior-level Full TimeUnited Kingdom3d ago
-
Intermediate Data Engineer GBP 40K-45KAWS | AWS RDS | Branching | CI/CD | CloudFormationAnnual hackathon events | Cycle to work scheme | Electric vehicle scheme | Flexible working | Health and wellbeing programmeMid-level Full TimeNorth West UK - Hybrid R3d ago
-
Data Engineer - AI/ML (f/m/d) GBP 50K-60KAWS | Agile | Azure | Azure Data | Azure Data FactoryFlexi-Week | Health and wellbeing support | Hybrid remote working | Peer to peer recognition voucher | Remote Working FurnitureMid-level Full TimeBerlin, Berlin, Germany; Iași, Iași, Romania; …3d ago