Senior Data Engineer
Telangana, India
⚠️ We'll shut down after Aug 1st - try foo🦍 for all jobs in tech ⚠️
Look candidates who can join quickly, within a 30-day timeframe
Experience 5-7 years
- Design and implement end-to-end data processing pipelines using Azure Data Factory and Databricks. This involves extracting data from various sources, transforming it to meet specific business requirements, and loading it into target data repositories.
- Integrate data from on-premises systems, third-party sources, and external APIs using Azure Data Factory. Develop efficient workflows and data transfer mechanisms to ensure high-quality, consistent, and reliable data ingestion.
- Optimize data processing pipelines and SQL queries to improve performance and reduce latency. Utilize query optimization techniques and data partitioning strategies to accelerate data processing.
- Implement and manage data repositories on Azure, such as Azure Data Lake Storage or Azure SQL Data Warehouse. Define and enforce data governance policies and security measures to protect sensitive data.
- Monitor data pipelines, jobs, and workflows to identify and resolve any issues or bottlenecks. Perform regular performance tuning, error handling, and troubleshooting to ensure smooth data processing.
- Stay updated with the latest advancements in Azure Data Factory and Databricks technologies. Continuously identify areas for improvement, automate repetitive processes, and enhance data engineering practices to drive efficiency and deliver high-quality data solutions.
- Collaborate with solution architects to design scalable and reliable data architectures in Azure, incorporating Azure Data Factory and Databricks components. Provide guidance on data modeling, data flow, and data integration patterns.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions. Document data pipelines, workflows, and technical specifications for future reference and knowledge sharing.
- Implement data governance and security best practices to ensure compliance with regulatory requirements and industry standards. Establish data access controls, data encryption, and data classification mechanisms.
- Stay up-to-date with industry trends to leverage advanced capabilities and optimize data engineering practices.
- Good to have knowledge in Snowflake.
Look candidates who can join quickly, within a 30-day timeframe
Experience 5-7 years
- Design and implement end-to-end data processing pipelines using Azure Data Factory and Databricks. This involves extracting data from various sources, transforming it to meet specific business requirements, and loading it into target data repositories.
- Integrate data from on-premises systems, third-party sources, and external APIs using Azure Data Factory. Develop efficient workflows and data transfer mechanisms to ensure high-quality, consistent, and reliable data ingestion.
- Optimize data processing pipelines and SQL queries to improve performance and reduce latency. Utilize query optimization techniques and data partitioning strategies to accelerate data processing.
- Implement and manage data repositories on Azure, such as Azure Data Lake Storage or Azure SQL Data Warehouse. Define and enforce data governance policies and security measures to protect sensitive data.
- Monitor data pipelines, jobs, and workflows to identify and resolve any issues or bottlenecks. Perform regular performance tuning, error handling, and troubleshooting to ensure smooth data processing.
- Stay updated with the latest advancements in Azure Data Factory and Databricks technologies. Continuously identify areas for improvement, automate repetitive processes, and enhance data engineering practices to drive efficiency and deliver high-quality data solutions.
- Collaborate with solution architects to design scalable and reliable data architectures in Azure, incorporating Azure Data Factory and Databricks components. Provide guidance on data modeling, data flow, and data integration patterns.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand data requirements and deliver solutions. Document data pipelines, workflows, and technical specifications for future reference and knowledge sharing.
- Implement data governance and security best practices to ensure compliance with regulatory requirements and industry standards. Establish data access controls, data encryption, and data classification mechanisms.
- Stay up-to-date with industry trends to leverage advanced capabilities and optimize data engineering practices.
- Good to have knowledge in Snowflake.
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Job stats:
1
0
0
Category:
Engineering Jobs
Tags: APIs Architecture Azure Classification Databricks Data governance Data pipelines Data warehouse Engineering Pipelines Security Snowflake SQL
Region:
Asia/Pacific
Country:
India
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.
Sr. Data Engineer jobsData Scientist II jobsBusiness Intelligence Developer jobsPrincipal Data Engineer jobsBI Developer jobsStaff Data Scientist jobsStaff Machine Learning Engineer jobsPrincipal Software Engineer jobsDevOps Engineer jobsJunior Data Analyst jobsData Science Intern jobsSoftware Engineer II jobsData Science Manager jobsData Manager jobsStaff Software Engineer jobsAI/ML Engineer jobsLead Data Analyst jobsData Analyst Intern jobsBusiness Data Analyst jobsSr. Data Scientist jobsData Specialist jobsData Engineer III jobsBusiness Intelligence Analyst jobsData Governance Analyst jobsData Analyst II jobs
Consulting jobsMLOps jobsAirflow jobsOpen Source jobsLinux jobsEconomics jobsKafka jobsKPIs jobsGitHub jobsJavaScript jobsTerraform jobsPostgreSQL jobsBanking jobsPrompt engineering jobsRAG jobsRDBMS jobsStreaming jobsNoSQL jobsPhysics jobsClassification jobsData Warehousing jobsComputer Vision jobsScikit-learn jobsdbt jobsGoogle Cloud jobs
GPT jobsLooker jobsHadoop jobsR&D jobsPandas jobsScala jobsData warehouse jobsLangChain jobsOracle jobsReact jobsDistributed Systems jobsBigQuery jobsMicroservices jobsELT jobsScrum jobsCX jobsPySpark jobsIndustrial jobsOpenAI jobsRedshift jobsJira jobsSAS jobsRobotics jobsTypeScript jobsE-commerce jobs