Lead Data Engineer - Azure Databricks/Kafka
Tasks
- Apply encryption and data masking
- Apply watermarking and primary key deduplication
- Automate deployments with CI/CD
- Build CDC and deduplication logic
- Deploy and update pipelines for many tables
- Design streaming ingestion pipelines
- Enforce role-based access control
- Ensure schema evolution
- Implement databricks auto loader
- Implement exactly once semantics
- Implement monitoring and observability
- Ingest data into Delta Lake
- Manage infrastructure with Infrastructure as Code
- Scale ingestion with config driven framework
- Set up dashboards for lag and processing metrics
- Use version control and code reviews
Perks/Benefits
- N/A
Skills/Tech-stack
Access Control | Airflow | Apache Kafka | Apache Spark | Auto Loader | Azure Event | Azure Event Hubs | CI/CD | Confluent Cloud | Data Lakehouse | Data masking | Databricks Auto Loader | Debezium | Delta Lake | Delta Live Tables | Delta Live) | Encryption | Event Hubs | GitHub Actions | Grafana | Java | Jenkins | Lakehouse | Oracle | Prometheus | Python | RabbitMQ | Role-Based Access | Role-Based Access Control | SQL | SQL Server | Scala | Schema evolution | Structured Streaming | Terraform | Watermarking
Education
N/A
Roles
Data Engineer | Engineer | Lead | Lead Data | Lead Data Engineer
Related jobs
-
Senior Full Stack Engineer AED 286K-323KAPI Gateway | AWS | Azure | Azure DocumentDB | Big DataCareer advancement opportunities | Certification support | Health insurance | Incentive program | Professional development supportSenior-level Full TimeAbu Dhabi, Abu Dhabi, United Arab …17h ago
-
Senior AI Engineer AED 286K-349KAI Studio | AWS Bedrock | Agent systems | Agentic Workflows | Asynchronous programmingCareer advancement | Employee incentive program | Health insurance | Professional development | Subscription ReimbursementSenior-level Full TimeAbu Dhabi, Abu Dhabi, United Arab …17h ago
-
Visualization Engineer AED 96K-120KContinuous Deployment | Continuous integration | Data Lake | Data Quality | Data WarehousingAccident insurance | Annual leave | Health insurance | Hybrid work | Life and medical coverMid-level Full TimeDubai - Dubai Media City, United …17h ago
-
Team Lead, Customer Analytics, Shipping Services AED 180K-240KAWS | BigQuery | Data Governance | Data Modeling | Data QualityCareer development opportunities | Measurable impact on business KPIs | Mentorship | Product lifecycle exposureSenior-level Full TimeDubai, Dubai, United Arab Emirates1d ago
-
Data Manipulation | Distributed Systems | Embeddings | Kubernetes | LLM InferenceCollaborative culture | Cutting edge generative AI exposure | Direct access to technical leadership | Flexible schedule | High autonomyEntry-level Full TimeUnited Arab Emirates R3d ago
-
Senior-level Full TimeDubai, Dubai, United Arab Emirates3d ago
-
Senior AI/ML Process Consultant/Engineer AED 286K-349KAI Governance | APC | Artificial Intelligence | Automation | Data GovernanceGlobal travel opportunities | Training and enablement supportSenior-level Full TimeYMA-UAE MUSAFFAH, United Arab Emirates3d ago
-
AI/ML Process Engineer AED 240K-240KAPC | Artificial Intelligence | Data Visualization | Database | Digital TwinGlobal travel opportunitiesSenior-level Full TimeYMA-UAE MUSAFFAH, United Arab Emirates3d ago
-
Senior-level Full TimeAbu Dhabi4d ago
-
Algorithms | Code review | Computer Vision | Data Preprocessing | Data StructuresMid-level Full TimeAbu Dhabi4d ago
-
Entry-level Full TimeAbu Dhabi4d ago
-
BeautifulSoup | Data Annotation | Data Pipelines | Data Preprocessing | Deep learningSenior-level Full TimeAbu Dhabi4d ago
-
Mid-level Full TimeAbu Dhabi4d ago
-
DevOps Engineer (6 months contract) AED 144K-168KAWS | Argo CD | Argo Rollouts | Azure | BashOn site work location | On-call rotationSenior-level ContractAbu Dhabi, United Arab Emirates6d ago
-
Data Engineer (6 months contract) AED 90K-96KAWS | Apache Airflow | Apache Spark | Azure | Azure DataMid-level ContractAbu Dhabi, United Arab Emirates6d ago
-
AI Engineer (6 months contract) AED 168K-180KAI Search | AWS | Agentic Workflows | Anthropic API | Apache SparkMid-level ContractAbu Dhabi, United Arab Emirates6d ago
-
Senior Data Engineer (Python) AED 300K-386KAPIs | Apache Airflow | Azure | Azure Data | Azure Data FactoryEmployment Visa | Flexible working environment | Hybrid work | Medical insuranceSenior-level Full TimeDubai, Dubai, United Arab Emirates (Hybrid) R6d ago
-
Analytics Engineer AED 120K-144KAmazon Redshift | DBT | Dashboards | Data Modeling | Data VisualizationEnd of service savings scheme | Equity compensation | Hybrid work | Learning and development budget | Off-sitesMid-level Full TimeDubai, United Arab Emirates6d ago
-
Senior-level Full TimeUnited Arab Emirates6d ago
-
Sr. Python Engineer - Microservices & Data Platforms AED 180K-228KAPI Gateway | AWS | Asyncio | Authentication | AuthorizationSenior-level Full TimeDubai, Dubai, United Arab Emirates6d ago
-
Nodejs with GenAI Developer (Arabic) AED 204K-228KAI Services | Async/Await | Asynchronous programming | Automation Testing | Azure AISenior-level Full TimeAE-DU-Dubai7d ago
-
Mid-level Full TimeAbu Dhabi, United Arab Emirates8d ago
-
Senior-level Full TimeUnited Arab Emirates9d ago
-
Forward Deployed Engineer - UAE AED 120K-120KBackend Development | Code review | Frontend Development | Full Stack | Full-Stack DevelopmentHybrid work model | Relocation assistance | Travel up to 50 percentMid-level Full TimeAbu Dhabi, UAE14d ago
-
Data Architect AED 156K-240KAirflow | Big Data | Cloud Computing | DBT | Data GovernanceCareer growth | Training and developmentSenior-level Full TimeAbu Dhabi, United Arab Emirates14d ago