Data Engineer 2

Introduction
At IBM, work is more than a job – it’s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better, but to attempt things you’ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world’s most challenging problems? If so, lets talk.

Your Role and Responsibilities

• We are seeking an experienced data engineer to join our team. As a data engineer, you will be responsible for designing, implementing, and maintaining data structures and pipelines for efficient data processing and analysis. Your role will involve working with various databases, including DB2, Cloudant, PostgreSQL, and REDIS. You will collaborate with cross-functional teams to understand data requirements, optimise data flow, and ensure data integrity. Your expertise in data modelling, ETL processes, and database technologies will be instrumental in building robust and scalable data solutions.
• Responsibilities:
• Design and implement data models and database structures to meet business requirements.
• Develop and maintain ETL processes to extract, transform, and load data from various sources.
• Optimise data pipelines and workflows for efficient data processing and analysis.
• Collaborate with data scientists and analysts to define data requirements and deliver actionable insights
• Ensure data quality, integrity, and security throughout the data lifecycle.
  Monitor and troubleshoot data pipelines, resolving issues and performance bottlenecks.
• Implement data governance and data management best practices.
  Stay updated with emerging trends and technologies in data engineering and database management.
• Collaborate with cross-functional teams to define data integration strategies and implement data-driven solutions.
• Implement data security measures and ensure compliance with relevant regulations and standards.
• Document data structures, processes, and workflows for knowledge sharing and future reference.
  Mentor junior data engineers and provide technical guidance and support.
• Tasks:
  Design and implement data models for DB2, Cloudant, PostgreSQL, and REDIS databases.
• Develop ETL processes using tools and frameworks such as Apache Spark, Apache Kafka, or Talend.
• Optimise data pipelines for performance, scalability, and reliability.
• Collaborate with data scientists and analysts to understand data requirements and develop data solutions.
• Implement data governance practices, including data quality monitoring and metadata management.
• Monitor and analyse data pipelines, identifying and resolving issues or bottlenecks.
• Implement data security measures, including access controls and encryption.
• Collaborate with database administrators to ensure optimal database performance and availability.
• Conduct data profiling and analysis to identify data quality issues and propose solutions.
  Implement and automate data validation and testing processes.
  Explore and evaluate new tools and technologies for data engineering and database management.

Required Technical and Professional Expertise

• Proven experience as a data engineer or similar role
• Strong expertise in designing data structures and database modelling
• Experience with DB2 and Cloudant databases is required.
• Knowledge of PostgreSQL and REDIS databases is a plus.
• Proficiency in ETL processes, Pentaho/DataStage, and tools such as Apache Spark, Apache Kafka, or Talend
• Familiarity with data warehousing concepts and technologies
• Strong SQL skills for data manipulation and querying
• Understanding of data governance and data management principles
• Knowledge of data security and compliance standards
• Experience with data integration and data pipeline optimisation
• Familiarity with cloud-based data platforms such as AWS or Azure
• Experience with data visualisation tools and techniques is a plus.

Preferred Technical and Professional Expertise

• strong analytical and problem-solving skills
• Excellent communication and collaboration abilities
• Ability to work in a fast-paced and dynamic environment
• Continuous learning mindset to stay updated with industry trends and advancements

Key Job Details
Role:Data Engineer 2 Location:Bangalore, IN Category:Data & Analytics Employment Type:Full-Time Travel Required:No Travel Contract Type:Regular Company:(0063) IBM India Private Limited Req ID:728479BR