Senior Data Engineer

We are seeking an experienced and highly skilled Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in building and optimizing large-scale data pipelines, architecting cloud-based data solutions, and managing complex data environments using cloud technologies. You will work closely with cross-functional teams to design, develop, and deploy data solutions that power business insights.

Key Responsibilities:

• Data Pipeline Development: Design, build, and maintain scalable ETL/ELT pipelines using AWS services such as Glue, EMR, and Athena. Leverage AWS Step Functions to automate complex workflows.
  
• Data Warehousing: Architect and manage data warehouses using cloud technologies, while integrating with S3, RDS, and DynamoDB for both structured (SQL) and unstructured (NoSQL) data needs.
  
• Big Data Processing: Utilize PySpark on AWS EMR for distributed data processing, ensuring the effective transformation of raw data into business insights.
  
• Database Management: Manage both SQL (e.g., PostgreSQL, MySQL) and NoSQL (e.g., DynamoDB, MongoDB) databases, ensuring data is efficiently stored, retrieved, and queried.
  
• Real-Time Data Processing: Implement and maintain real-time streaming data solutions using AWS Kinesis or Kafka for low-latency data processing.
  
• Cloud Monitoring & Automation: Use AWS CLI and CloudWatch to monitor, automate, and optimize data pipelines and ensure efficient use of cloud resources.
  
• Data Governance & Security: Ensure adherence to best practices for data security, governance, and compliance, including data encryption and access controls.
  
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to align data infrastructure with business objectives.
  
• Documentation: Maintain detailed documentation of data models, data pipeline processes, and system architectures.
  

Requirements

• 5+ years of experience as a Data Engineer, with a strong emphasis on cloud-based data warehousing and ETL pipelines.
  
• Proficiency with AWS services including EMR, Athena, Glue, Step Functions, ElasticSearch, RDS, Redshift, DynamoDB, and S3.
  
• Advanced SQL skills and experience with relational databases like PostgreSQL, MySQL as well as NoSQL databases such as MongoDB and DynamoDB.
  
• Hands-on experience in Python for data processing and automation, with knowledge of AWS CLI.
  
• Strong proficiency with PySpark for distributed data processing and transformation.
  
• Experience with AWS CloudWatch for monitoring and optimizing data workflows.
  
• Data warehouse architecture experience, including optimizing query performance in large-scale environments.
  
• Strong understanding of data governance, security, and compliance best practices.
  
• Bachelor's degree in Computer Science, Engineering, or a related field.
  

Nice to have:

• Experience with Azure Data Technologies (e.g., Azure Data Lake, Azure Synapse, Azure Databricks).
  
• Familiarity with real-time data streaming solutions such as Apache Kafka or AWS Kinesis.
  
• AWS Big Data Certified.