Lead Data Engineer - Azure Databricks/Kafka
Tasks
- Apply encryption and data masking
- Apply watermarking and primary key deduplication
- Automate deployments with CI/CD
- Build CDC and deduplication logic
- Deploy and update pipelines for many tables
- Design streaming ingestion pipelines
- Enforce role-based access control
- Ensure schema evolution
- Implement databricks auto loader
- Implement exactly once semantics
- Implement monitoring and observability
- Ingest data into Delta Lake
- Manage infrastructure with Infrastructure as Code
- Scale ingestion with config driven framework
- Set up dashboards for lag and processing metrics
- Use version control and code reviews
Perks/Benefits
- N/A
Skills/Tech-stack
Access Control | Airflow | Apache Kafka | Apache Spark | Auto Loader | Azure Event | Azure Event Hubs | CI/CD | Confluent Cloud | Data Lakehouse | Data masking | Databricks Auto Loader | Debezium | Delta Lake | Delta Live Tables | Delta Live) | Encryption | Event Hubs | GitHub Actions | Grafana | Java | Jenkins | Lakehouse | Oracle | Prometheus | Python | RabbitMQ | Role-Based Access | Role-Based Access Control | SQL | SQL Server | Scala | Schema evolution | Structured Streaming | Terraform | Watermarking
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Engineer, Data & Agentic AI AED 120K-144KCloud Computing | Data Pipelines | Deep learning | ETL | Language ProcessingTraining and developmentMid-level Full TimeAbu Dhabi, United Arab Emirates1d ago
-
Senior-level Full TimeDubai, Dubai, United Arab Emirates1d ago
-
Data Architect - Treasury AED 240K-323KAccess Control | Amazon Web Services | Azure Synapse | Azure Synapse Analytics | Batch Data IntegrationCareer advancement | Certification support | Employee incentive program | Health insurance | Professional development supportSenior-level Contract Full TimeDubai, Dubai, United Arab Emirates1d ago
-
Agentic DevOps Lead – Data & AI AED 264K-323KAWS | Autogen | CI/CD | Cloud Architecture | Cloud platformCareer progression | Competitive rewards | Continuous coaching | Flexible work arrangements | Travel as neededSenior-level Full TimeDubai, Arenco Tower, United Arab Emirates1d ago
-
Mid-level Full TimeDubai, United Arab Emirates2d ago
-
Manager Crew Insights & Optimization AED 120K-144KCloud Data | Cloud Data Warehouse | Data Warehouse | Databricks | Descriptive AnalyticsMid-level Full TimeAbu Dhabi, United Arab Emirates2d ago
-
Deep learning | Generative AI | Language Models | Language Processing | Large Language ModelsMid-level Full TimeAbu Dhabi, United Arab Emirates2d ago
-
Alerting | Amazon Redshift | Batch Processing | CI/CD | Cause analysisCollaboration with cross-functional teams | Fully remote work | Ownership over data reliability strategy | Remote opportunitySenior-level Full TimeUnited Arab Emirates R2d ago
-
Generative AI / Azure AI Engineering Lead AED 192K-234KACL | AI Foundry | AI Search | API Security | Agent systemsSenior-level Full TimeDubai, Arenco Tower, United Arab Emirates2d ago
-
Data Warehouse Engineer - Intern AED 132K-198KApache Spark | Azure | Azure Data | Azure Data Factory | Data FactoryEmployee wellbeing | Growth opportunitiesEntry-level Full Time InternshipUAE Dubai (PIL Office), United Arab …2d ago
-
Algorithms | Data Science | Machine Learning | Python | Software EngineeringInternship experienceEntry-level InternshipDubai, Dubai, United Arab Emirates2d ago
-
Senior-level Full TimeDubai, United Arab Emirates5d ago
-
Senior Data Engineer - Python & Microservices AED 295K-349KAPI Security | AWS | Apache Kafka | Apache Spark | Asynchronous programmingSenior-level Full TimeDubai, Dubai, United Arab Emirates7d ago
-
Data Engineer AED 144K-156KAccess Management | Amazon Redshift | Azure | Azure Data | Azure Data FactoryMid-level Full TimeUnited Arab Emirates8d ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Friendly atmosphere | Portfolio and LinkedIn application opportunity | Supportive management approachMid-level Full TimeDubai, United Arab Emirates8d ago
-
Senior AI Solutions Engineer - UAE AED 286K-360KAI Evaluation | AI Observability | AI tracing | APIs | Agent OrchestrationDental insurance | Employee assistance program | Flexible time off | Medical insurance | Paid HolidaysSenior-level Full TimeRemote Dubai, United Arab Emirates R8d ago
-
Snowflake Data Engineer AED 200K-200KAPI | Access Control | Cloud Data | Cloud Data Storage | Data GovernanceMid-level Full TimeDubai, AE9d ago
-
AWS | Apache Kafka | Azure | CI/CD | Confluent Control CenterMid-level Full TimeAbu Dhabi, United Arab Emirates13d ago
-
Senior Research Engineer in the Division of Science (Computer Science) - Dr. Azza Abouzied AED 280K-323KArtificial Intelligence | C++ | Data Visualization | Database systems | DockerAnnual home leave allowance | Computing support | Dependent private school fees | Educational assistance | Housing allowanceSenior-level Full TimeAbu Dhabi, UAE13d ago
-
Lead Data Engineer AED 168K-222KApache Airflow | Astronomer | Azure | Azure Data | Azure Data FactorySenior-level Full Timedubai, Dubai, United Arab Emirates14d ago
-
AI Engineer AED 102K-102KAWS | Azure | Generative AI | Google Cloud | Image SynthesisDocumentation support | Hands on troubleshooting support | Technical training sessions | Workshops and demosMid-level Full TimeDubai, United Arab Emirates14d ago
-
AI Data Engineer AED 300K-360KAWS | Apache Airflow | Azure | BigQuery | CI/CDAccommodation | Expense reimbursements for home visits | Flight tickets | Holiday bonuses | IPhoneMid-level Full TimeDubai, United Arab Emirates14d ago
-
Data Engineer - Automation & AI AED 180K-228KAI Agents | AI Foundry | AWS Bedrock | AWS Glue | Apache SparkSenior-level Full TimeAbu Dhabi, Abu Dhabi, United Arab …14d ago
-
AWS Lakehouse Platform Engineer AED 180K-228KAWS CDK | AWS Glue | AWS Lake Formation | AWS S3 | AWS Well ArchitectedMentoring | On-site workSenior-level Full TimeAbu Dhabi, Abu Dhabi, United Arab …14d ago
-
Senior Data Scientist II - Personalization AED 120K-144KA/B | A/B Testing | Apache Spark | Attention Mechanism | B testingFitness reimbursements | Healthcare benefits | Remote work 30 days per year | Training classes access | Unlimited vacation daysMid-level Full TimeDubai, United Arab Emirates R15d ago