Data Engineering, Consultant
MY-Kuala Lumpur-Menara AIA, Malaysia
USD 119K-180K (estimate) Senior-level Full Time
Tasks
- Act as technical lead for data engineering initiatives
- Adhere to Azure service provisioning access management and security compliance
- Assess emerging data engineering technologies for enterprise applicability
- Build and operate Azure storage and data factory pipelines with ADLS Gen2 ADF and Azure DevOps
- Build data pipelines using PySpark
- Champion best practices for data engineering cloud architecture and AI integration
- Collaborate with application teams on delivery integration requirements and timelines
- Contribute to enterprise standards and reusable components
- Define business as usual support model and manage BAU operations
- Design build and optimize scalable data pipelines and architectures
- Develop Databricks workflows and cluster operations with notebooks and jobs
- Develop complex SQL queries including joins window functions and data modeling
- Ensure governance and compliance for services and infrastructure provisioning
- Implement Delta Lake features including merge upsert time travel partitioning optimize and vacuum
- Implement data governance with data lineage and access control
- Implement lakehouse architecture bronze silver gold ETL ELT design
- Implement streaming with Auto Loader Structured Streaming and Kafka or Event Hub
- Integrate API services with enterprise data platforms
- Manage access and user governance with Unity Catalog
- Manage vendor relationships for quality security and performance
- Oversee end to end project execution from design through deployment
- Resolve operational issues and drive continuous improvement
- Set up CI CD pipelines with Terraform and Databricks Asset Bundles
Perks/Benefits
- N/A
Skills/Tech-stack
ADLS Gen2 | API Integration | Access Control | Apache Spark | Asset bundles | Auto Loader | Azure | Azure Data | Azure Data Factory | Azure Databricks | Azure DevOps | CI/CD | Data Factory | Data Lakehouse | Data Lineage | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Workflows | Delta Lake | ELT | ETL | Event Hubs | Git | Kafka | Notebooks | PySpark | SQL | Structured Streaming | Terraform | Unity Catalog
Education
Regions
Countries
States
Related jobs
-
Software Development Engineer USD 115K-195KAPI Development | AWS | Agile | Artificial Intelligence | Automated testingDiversity and inclusion | Hybrid working | Professional growthSenior-level Full TimeMYS, Petaling Jaya (NDMY), Malaysia2d ago
-
Associate AI Solutions Engineer USD 88K-152KAPI Development | Agile | C# | C++ | CI/CDHybrid work environmentMid-level Full TimeMYS, Petaling Jaya (NDMY), Malaysia2d ago
-
Junior Data Engineer EUR 26K-39KAWS | Accounts Payable | Accounts Receivable | Azure | Balance sheetEntry-level Full TimeKuala Lumpur, Malaysia3d ago
-
Senior Software Engineer – Enterprise AI Solutions USD 115K-207KAPIs | AWS | Agile | Algorithms | Artificial IntelligenceHybrid work environmentSenior-level Full TimeMYS, Petaling Jaya (NDMY), Malaysia3d ago
-
Bash | Cloud infrastructure | Data Processing | Docker | GCPAsynchronous culture | Career growth impact | Inclusive mission | Laid back environment | Remote-friendly teamMid-level Full TimePenang, Malaysia4d ago
-
Senior-level Full TimeMalaysia4d ago
-
Senior Executive, Data Engineer USD 119K-160KArtificial Intelligence | Cloud Data | Cloud Data Platforms | Data Lake | Data MartSenior-level Full TimeSD Guthrie Tower, Malaysia4d ago
-
Data Science Engagement Lead USD 124K-186KAI/ML | AWS | Agile | Artificial Intelligence | AzureFlexible working hours | Paid parental leave | Remote work optionSenior-level Full TimeShell Centre Kuala Lumpur, Malaysia4d ago
-
Senior Developer, Machine Learning Operations USD 149K-200KAPI Integration | Agentic AI | Amazon Web Services | Artifactory | BashSenior-level Full TimeThe Gardens North Tower, Malaysia4d ago
-
Yield Analytics Engineer USD 120K-194KData analytics | JavaScript | Machine Learning | Perl | PythonOnsite workMid-level Full TimeMYS - Penang, Malaysia4d ago
-
Engineer, Manufacturing Engineering USD 81K-108KData Modeling | Generative AI | Machine Learning | Python | SQLDevelopment curriculum | Mentorship | Structured onboardingEntry-level Full TimeBayan Lepas, Penang, Malaysia5d ago
-
Head of Analytics & Market Intelligence USD 178K-240KAnalytics Strategy | Dashboard Development | Data Exploration | Data Governance | Data ManagementCompany car or car allowance | Employee assistance program | Flexible working | Health care | Learning and developmentExecutive-level Full TimeMalaysia - Kuala Lumpur5d ago
-
Mid-level Full TimeKuala Lumpur, MY5d ago
-
Mid-level Full TimeKuala Lumpur, Exchange 106, Malaysia5d ago
-
Senior Data Engineer CAD 142K-190KA/B | A/B Testing | Apache Airflow | B testing | Cloud DataHealth insurance | Hybrid work flexibility | Learning budget | Performance bonusesSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …6d ago
-
Digital Solution Engineer USD 119K-261KAWS | Amazon Redshift | Azure SQL | Big Data | Cloud DataFlexible workSenior-level Full TimeMalaysia, Federal Territory of Kuala Lumpur, …6d ago
-
Senior Data Engineer CAD 142K-190KA/B | A/B Testing | Apache Airflow | B testing | Data GovernanceHealth insurance | Hybrid work flexibility | Learning budget | Performance bonusesSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …6d ago
-
Data Analysis | Data Modeling | Data Pipelines | Data Transformation | PythonCareer development | Hybrid workMid-level Full TimeKuala Lumpur, Federal Territory of Kuala …7d ago
-
Senior Data Analytics Engineer USD 132K-203KAdvanced Process Control | Data Analysis | Exploratory Data Analysis | Fault detection | Industrial IoTSenior-level Full TimeBatu Kawan, Penang, Malaysia9d ago
-
Automation | Autosys | Batch Processing | Cause analysis | Data PipelinesMid-level Full TimeMY-Kuala Lumpur-IT, Malaysia9d ago
-
Autosys | Batch scheduling | Cause analysis | Data Pipelines | Data QualityMid-level Full TimeMY-Kuala Lumpur-IT, Malaysia9d ago
-
AWS | Azure | C# | C++ | CI/CDHybrid work scheduleSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …9d ago
-
C# | C++ | DHCP | Driver Development | Embedded LinuxFlexible home-office days | Hybrid work | Office in KL City | Office in KL SouthSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …9d ago
-
Senior-level Full TimePetaling Jaya (First Avenue), Malaysia10d ago
-
C# | C++ | DHCP | Docker | Embedded LinuxFlexible work location | Hybrid work schedule | Onsite office accessSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …10d ago