Data Engineering, Consultant
MY-Kuala Lumpur-Menara AIA, Malaysia
USD 119K-180K (estimate) Senior-level Full Time
Tasks
- Act as technical lead for data engineering initiatives
- Adhere to Azure service provisioning access management and security compliance
- Assess emerging data engineering technologies for enterprise applicability
- Build and operate Azure storage and data factory pipelines with ADLS Gen2 ADF and Azure DevOps
- Build data pipelines using PySpark
- Champion best practices for data engineering cloud architecture and AI integration
- Collaborate with application teams on delivery integration requirements and timelines
- Contribute to enterprise standards and reusable components
- Define business as usual support model and manage BAU operations
- Design build and optimize scalable data pipelines and architectures
- Develop Databricks workflows and cluster operations with notebooks and jobs
- Develop complex SQL queries including joins window functions and data modeling
- Ensure governance and compliance for services and infrastructure provisioning
- Implement Delta Lake features including merge upsert time travel partitioning optimize and vacuum
- Implement data governance with data lineage and access control
- Implement lakehouse architecture bronze silver gold ETL ELT design
- Implement streaming with Auto Loader Structured Streaming and Kafka or Event Hub
- Integrate API services with enterprise data platforms
- Manage access and user governance with Unity Catalog
- Manage vendor relationships for quality security and performance
- Oversee end to end project execution from design through deployment
- Resolve operational issues and drive continuous improvement
- Set up CI CD pipelines with Terraform and Databricks Asset Bundles
Perks/Benefits
- N/A
Skills/Tech-stack
ADLS Gen2 | API Integration | Access Control | Apache Spark | Asset bundles | Auto Loader | Azure | Azure Data | Azure Data Factory | Azure Databricks | Azure DevOps | CI/CD | Data Factory | Data Lakehouse | Data Lineage | Databricks | Databricks (asset) | Databricks Asset Bundles | Databricks Workflows | Delta Lake | ELT | ETL | Event Hubs | Git | Kafka | Notebooks | PySpark | SQL | Structured Streaming | Terraform | Unity Catalog
Education
Regions
Countries
States
Related jobs
-
Mid-level Full TimeMelaka12h ago
-
Big Data Development Engineer USD 156K-218KApache Doris | Apache Flink | Apache Kafka | Apache Spark | Business IntelligenceCareer advancement | Global collaboration | Internal events | Internal mobility | Study Growth FundEntry-level Full TimeKuala Lumpur, Malaysia1d ago
-
Intern USD 40K-50KApps Script | Data Modeling | Database normalization | Google Apps | Google Apps ScriptMentorship | Real-world project experienceEntry-level Full Time InternshipKuala Lumpur - RedQ, Malaysia1d ago
-
Senior Software Engineer, Backend (Fulfillment) USD 175K-230KAgile | Best practices | Git | Go | GolangBirthday leave | Custom benefits | Employee assistance program | Flexible work | Medical insuranceSenior-level Full TimePetaling Jaya, Malaysia2d ago
-
Entry-level Contract Full TimeMid Valley City, KUALA LUMPUR, Malaysia2d ago
-
Senior Data Engineer USD 109K-161KAWS | AWS Glue | AWS Lambda | AWS Step Functions | Amazon AthenaCareer development | Flexible working arrangements | Hybrid work model | Paid parental leave | Work-life balanceSenior-level Full TimeMY: Kuala Lumpur - Bangsar South …2d ago
-
Digital Solution Engineer USD 119K-261KAmazon Redshift | Azure Cosmos | Azure Cosmos DB | Azure SQL | Azure SynapseContinuous learning | Flexible work opportunities | Immersive learningSenior-level Full TimeMalaysia, Federal Territory of Kuala Lumpur, …3d ago
-
Senior Data Engineer USD 160K-200KAccess Control | BI platform | Backup and Restore | Batch Processing | CDCSenior-level Full TimeKuala Lumpur, Federal Territory of Kuala …3d ago
-
Azure Infrastructure & Data Engineer USD 125K-185KAKS | ARM | App Service | Application Gateway | AzureMid-level Full TimeMY - PENANG 38 JALAN SULTAN …3d ago
-
Azure Infrastructure & Data Engineer USD 125K-170KAKS | Access Control | App Service | Application Gateway | AzureContinuing education program | Continuous learning resources | Family-friendly perks | Financial wellness programs | Generous time offMid-level Full TimeMY - PENANG 38 JALAN SULTAN …3d ago
-
Senior-level Full TimeGeorge Town, Penang, Malaysia4d ago
-
Software Engineer II USD 100K-158KAWS | Agile | Angular | Azure | BitbucketHybrid work arrangementMid-level Full TimeCyberjaya, Selangor, Malaysia R4d ago
-
BI Engineer - IT GBP 42K-48KAzure Synapse | Azure Synapse Analytics | Data Governance | Data Management | Data ModelingMid-level Full TimeKuala Lumpur, Federal Territory of Kuala …4d ago
-
Software Engineer I EUR 38K-64KAPIs | AWS | Agile | Angular | CI/CDDiscretionary bonus plan | Hybrid work environmentMid-level Full TimeCyberjaya, Selangor, Malaysia R5d ago
-
Software System Engineer 1 USD 84K-99KCode Quality | Data Pipelines | Development Lifecycle | ETL | Embedded SystemsEntry-level Full TimeBukit Mertajam, Pulau Pinang, Malaysia7d ago
-
Manager, Data & Analytics (Advertising Technology) USD 60K-140KAmazon Web Services | Attribution Modeling | Azure | BigQuery | Clean Room technologyMid-level Full TimeKuala Lumpur, Malaysia8d ago
-
Machine Learning Engineer EUR 45K-60KAWS | Argo | BigQuery | Cloud Computing | DockerEmployee Assistance Program (EAP) | Flexible working environment | Hybrid work | LinkedIn Learning | Ongoing trainingsMid-level Full TimeKuala Lumpur, 14, Malaysia R9d ago
-
Data Engineer USD 149K-175KAgile | Airflow | Amazon QuickSight | Amazon Redshift | Amazon Web ServicesCareer coaching | Coaching | Employee assistance programme | Flexible working hours | Hybrid workSenior-level Full TimeKuala Lumpur, Kuala Lumpur, Malaysia R9d ago
-
Entry-level Full TimePetaling Jaya, Selangor, Malaisie10d ago
-
Mid-level Full TimePetaling Jaya, Selangor, Malaisie10d ago
-
Full Stack AI Solution Engineer USD 38KAWS | Docker | JavaScript | Kubernetes | Language ProcessingMid-level Full TimePetaling Jaya, Selangor, Malaisie10d ago
-
Mid-level Full TimeWilayah Persekutuan Kuala Lumpur, Malaysia10d ago
-
Lead Data Engineer USD 171K-214KAWS | AWS Step Functions | Airflow | Amazon Redshift | CDCAI assisted development environment | Career growth mentoring | On-call rotationSenior-level Full TimeKuala Lumpur, Wilayah Persekutuan Kuala Lumpur, …10d ago
-
Data Warehouse Engineer USD 120K-170KData Warehouse | Data Warehouse Architecture | Doris | ETL | FlinkCareer advancement | Global collaboration | Internal events | Internal mobility | Study Growth FundEntry-level Full TimeKuala Lumpur, Malaysia10d ago
-
Machine Learning Engineer INR 2500K-3000KAzure | Business Intelligence | Data Warehousing | Data analytics | Data platformMid-level Full TimeKuala Lumpur, Exchange 106, Malaysia10d ago