Senior Data Engineer
Telangana, India
⚠️ We'll shut down after Aug 1st - try foo🦍 for all jobs in tech ⚠️
Look candidates who can join quickly, within a 30-day timeframe
Experience 5-7 years
- Design and implement end-to-end data processing pipelines using Azure Data Factory and Databricks. This involves extracting data from various sources, transforming it to meet specific business requirements, and loading it into target data repositories.
- Integrate data from on-premises systems, third-party sources, and external APIs using Azure Data Factory. Develop efficient workflows and data transfer mechanisms to ensure high-quality, consistent, and reliable data ingestion.
- Optimize data processing pipelines and SQL queries to improve performance and reduce latency. Utilize query optimization techniques and data partitioning strategies to accelerate data processing.
- Implement and manage data repositories on Azure, such as Azure Data Lake Storage or Azure SQL Data Warehouse. Define and enforce data governance policies and security measures to protect sensitive data.
- Monitor data pipelines, jobs, and workflows to identify and resolve any issues or bottlenecks. Perform regular performance tuning, error handling, and troubleshooting to ensure smooth data processing.
- Stay updated with the latest advancements in Azure Data Factory and Databricks technologies. Continuously identify areas for improvement, automate repetitive processes, and enhance data engineering practices to drive efficiency and deliver high-quality data solutions.
- Collaborate with solution architects to design scalable and reliable data architectures in Azure, incorporating Azure Data Factory and Databricks components. Provide guidance on data modeling, data flow, and data integration patterns.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions. Document data pipelines, workflows, and technical specifications for future reference and knowledge sharing.
- Implement data governance and security best practices to ensure compliance with regulatory requirements and industry standards. Establish data access controls, data encryption, and data classification mechanisms.
- Stay up-to-date with industry trends to leverage advanced capabilities and optimize data engineering practices.
- Good to have knowledge in Snowflake.
Look candidates who can join quickly, within a 30-day timeframe
Experience 5-7 years
- Design and implement end-to-end data processing pipelines using Azure Data Factory and Databricks. This involves extracting data from various sources, transforming it to meet specific business requirements, and loading it into target data repositories.
- Integrate data from on-premises systems, third-party sources, and external APIs using Azure Data Factory. Develop efficient workflows and data transfer mechanisms to ensure high-quality, consistent, and reliable data ingestion.
- Optimize data processing pipelines and SQL queries to improve performance and reduce latency. Utilize query optimization techniques and data partitioning strategies to accelerate data processing.
- Implement and manage data repositories on Azure, such as Azure Data Lake Storage or Azure SQL Data Warehouse. Define and enforce data governance policies and security measures to protect sensitive data.
- Monitor data pipelines, jobs, and workflows to identify and resolve any issues or bottlenecks. Perform regular performance tuning, error handling, and troubleshooting to ensure smooth data processing.
- Stay updated with the latest advancements in Azure Data Factory and Databricks technologies. Continuously identify areas for improvement, automate repetitive processes, and enhance data engineering practices to drive efficiency and deliver high-quality data solutions.
- Collaborate with solution architects to design scalable and reliable data architectures in Azure, incorporating Azure Data Factory and Databricks components. Provide guidance on data modeling, data flow, and data integration patterns.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions. Document data pipelines, workflows, and technical specifications for future reference and knowledge sharing.
- Implement data governance and security best practices to ensure compliance with regulatory requirements and industry standards. Establish data access controls, data encryption, and data classification mechanisms.
- Stay up-to-date with industry trends to leverage advanced capabilities and optimize data engineering practices.
- Good to have knowledge in Snowflake.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Job stats:
1
0
0
Category:
Engineering Jobs
Tags: APIs Architecture Azure Classification Databricks Data governance Data pipelines Data warehouse Engineering Pipelines Security Snowflake SQL
Region:
Asia/Pacific
Country:
India
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.
Sr. Data Engineer jobsData Scientist II jobsBI Developer jobsPrincipal Data Engineer jobsBusiness Intelligence Developer jobsStaff Data Scientist jobsStaff Machine Learning Engineer jobsPrincipal Software Engineer jobsJunior Data Analyst jobsData Science Intern jobsDevOps Engineer jobsData Manager jobsSoftware Engineer II jobsData Science Manager jobsStaff Software Engineer jobsAI/ML Engineer jobsData Analyst Intern jobsLead Data Analyst jobsBusiness Data Analyst jobsSr. Data Scientist jobsData Specialist jobsData Engineer III jobsBusiness Intelligence Analyst jobsSenior Backend Engineer jobsData Governance Analyst jobs
Consulting jobsMLOps jobsAirflow jobsOpen Source jobsEconomics jobsLinux jobsKafka jobsKPIs jobsGitHub jobsTerraform jobsJavaScript jobsPostgreSQL jobsPrompt engineering jobsRDBMS jobsBanking jobsNoSQL jobsComputer Vision jobsClassification jobsStreaming jobsData Warehousing jobsRAG jobsScikit-learn jobsPhysics jobsGoogle Cloud jobsdbt jobs
Hadoop jobsGPT jobsPandas jobsLooker jobsBigQuery jobsData warehouse jobsScala jobsLangChain jobsReact jobsR&D jobsDistributed Systems jobsOracle jobsMicroservices jobsScrum jobsELT jobsCX jobsPySpark jobsIndustrial jobsOpenAI jobsRedshift jobsTypeScript jobsJira jobsSAS jobsModel training jobsRobotics jobs