Data Engineering, Consultant
MY-Kuala Lumpur-Menara AIA, Malaysia
USD 119K-180K (estimate) Senior-level Full Time
Tasks
- Act as technical lead for data engineering initiatives
- Adhere to Azure service provisioning access management and security compliance
- Assess emerging data engineering technologies for enterprise applicability
- Build and operate Azure storage and data factory pipelines with ADLS Gen2 ADF and Azure DevOps
- Build data pipelines using PySpark
- Champion best practices for data engineering cloud architecture and AI integration
- Collaborate with application teams on delivery integration requirements and timelines
- Contribute to enterprise standards and reusable components
- Define business as usual support model and manage BAU operations
- Design build and optimize scalable data pipelines and architectures
- Develop Databricks workflows and cluster operations with notebooks and jobs
- Develop complex SQL queries including joins window functions and data modeling
- Ensure governance and compliance for services and infrastructure provisioning
- Implement Delta Lake features including merge upsert time travel partitioning optimize and vacuum
- Implement data governance with data lineage and access control
- Implement lakehouse architecture bronze silver gold ETL ELT design
- Implement streaming with Auto Loader Structured Streaming and Kafka or Event Hub
- Integrate API services with enterprise data platforms
- Manage access and user governance with Unity Catalog
- Manage vendor relationships for quality security and performance
- Oversee end to end project execution from design through deployment
- Resolve operational issues and drive continuous improvement
- Set up CI CD pipelines with Terraform and Databricks Asset Bundles
Perks/Benefits
- N/A
Skills/Tech-stack
ADLS Gen2 | API Integration | Access Control | Apache Spark | Asset bundles | Auto Loader | Azure | Azure Data | Azure Data Factory | Azure Databricks | Azure DevOps | CI/CD | Data Factory | Data Lakehouse | Data Lineage | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Workflows | Delta Lake | ELT | ETL | Event Hubs | Git | Kafka | Notebooks | PySpark | SQL | Structured Streaming | Terraform | Unity Catalog
Education
Regions
Countries
States
Related jobs
-
Senior Data Engineer CAD 140K-185KAWS | Algorithms | Apache Airflow | Apache Spark | Auto ScalingBirthday leave | Employee assistance programme | FlexWork | GrabFlex | Medical insuranceSenior-level Full TimePetaling Jaya, Malaysia1d ago
-
Mid-level Full TimeKuala Lumpur - RedQ, Malaysia1d ago
-
Senior Data Engineer USD 98K-157KAPI Integration | Azure Data | Azure Data Factory | Azure SQL | Azure SQL DatabaseFlexible work practices | Hybrid work | Insurance coverage | Learning and development | Medical coverageSenior-level Full TimePetaling Jaya, Malaysia1d ago
-
Data platform technical lead USD 115K-183KApache Storm | Automation | Azure | Azure Data Factory | BashDiversity and inclusion | Hybrid workingSenior-level Full TimeMYS, Petaling Jaya (NDMY), Malaysia1d ago
-
Access Control | Backup and Recovery | Clustering | Database Backup | Database backup and recoveryMid-level Full TimeKuala Lumpur, Malaysia1d ago
-
Agile | Batch data | Batch data ingestion | Data Ingestion | Data QualityMid-level Full TimeKuala Lumpur, Malaysia1d ago
-
Agile | Batch data | Batch data ingestion | Data Ingestion | Data ModelingMid-level Full TimeKuala Lumpur, Malaysia1d ago
-
Azure Data | Azure Data Factory | BaaN | Data Factory | Data ModelingMid-level Full TimeBayan Lepas, Penang, Malaysia1d ago
-
Data Engineer PLN 150K-276KAgile | Authentication | Azure | Azure Functions | DAXAnnual leave | Birthday leave | Career progression | Company pension | Employee assistance programMid-level Full TimeKuala Lumpur, Malaysia1d ago
-
Lead Software Engineer GBP 32K-35KAWS | AWS Lambda | Amazon ECS | Amazon EKS | Auto ScalingHybrid work | Remote work flexibility | Reward and recognition | Volunteering opportunities | Wellness programsEntry-level Full TimeCyberjaya, Selangor, Malaysia R2d ago
-
Software Engineer I GBP 32K-33KAWS | Angular | CI/CD | Git | JavaEmployee benefits | Hybrid work arrangementEntry-level Full TimeCyberjaya, Selangor, Malaysia2d ago
-
Advanced Analytics | Big Data | Data Management | Data Quality | Data pipelineEntry-level Full TimeLocation(s): Kuala Lumpur, Kuala Lumpur, Malaysia2d ago
-
Cloud Consultant, Data Analytics USD 149K-220KAI machine learning | Apache Spark | Cassandra | Cloud Data | Cloud Data MigrationSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …2d ago
-
Engineer, Data Analytics Engineering USD 119K-258KAWS | Azure | Data Engineering | Data Governance | Data QualityMid-level Full TimeBatu Kawan, Penang, Malaysia2d ago
-
Senior Data Governance Engineer USD 128K-162KAmazon Web Services | Automation | Azure | Business glossary | CI/CDSenior-level Full TimeKuala Lumpur, MY2d ago
-
Engineer, Data Analytics Engineering USD 119K-258KAWS | Azure | Big Data | Cloud Computing | Data pipelineMid-level Full TimeBatu Kawan, Penang, Malaysia2d ago
-
Senior Data Analytics Quality Engineer USD 156K-190K8D | Artificial Intelligence | Automated testing | Cloud Computing | Data AnalysisFlexible work environment | Remote work optionSenior-level Full TimeShah Alam, Malaysia2d ago
-
Senior Data Analytics Quality Engineer USD 156K-190K8D Problem Solving | Agile | Artificial Intelligence | Automated testing | Cloud ComputingEmployee resource groups | Flexible work environment | Remote work optionSenior-level Full TimeShah Alam, Malaysia2d ago
-
Senior Data Analytics Quality Engineer USD 156K-190K5-why | 8D | Agile | Automated testing | Data GovernanceFlexible work environment | Remote work optionSenior-level Full TimeShah Alam, Malaysia2d ago
-
Data Analytics Quality Engineering Manager USD 162K-200K5 Whys | 8D methodology | Agile | Artificial Intelligence | Automated testingEmployee resource groups | Flexible work environment | Inclusive work environment | Remote work optionSenior-level Full TimeIpoh, Perak, Malaysia2d ago
-
Data Analytics Quality Engineering Manager USD 162K-200K5 Whys | 8D | Agile | Artificial Intelligence | Automated testingFlexible work environment | Remote work optionSenior-level Full TimeBayan Lepas, Penang, Malaysia2d ago
-
Senior Data Analytics Quality Engineer USD 156K-190K5 Whys | 8D | Agile | Artificial Intelligence | Automated testingEmployee resource groups | Flexible work environment | Remote work option | Rotational weekend supportSenior-level Full TimeIpoh, Perak, Malaysia3d ago
-
Senior Data Analytics Quality Engineer USD 156K-190K5 Whys | 8D | Agile | Data Governance | Data ModelingFlexible work environment | Remote work optionSenior-level Full TimeShah Alam, Malaysia3d ago
-
Customer Engineer (AIx Data Analytics-Enabled) USD 74K-102KAI | Anomaly Detection | Corrective Maintenance | Data Visualization | Data analyticsEmployee wellbeing programs | Supportive work culture | Training and developmentEntry-level Full TimeKulim,MYS, Malaysia3d ago
-
Senior Data Engineer PLN 206K-328KBig Data | Big-data frameworks | Cloud platform | Confluence | ContainerizationSenior-level Full TimeKuala Lumpur - RedQ, Malaysia3d ago