Program Manager

Job Description: AWS Tech Lead with Databricks Knowledge

Overview:

An AWS Tech Lead with Databricks SME (Subject Matter Expert) knowledge is responsible for designing, deploying, and managing cloud-based infrastructure on AWS and supporting data solutions using Databricks. The role involves leading a team to ensure scalable, secure, and cost-efficient cloud services while integrating data pipelines and analytics solutions.

Roles and Responsibilities:

• Cloud Infrastructure Management:
  • Design, implement, and maintain AWS cloud solutions, including VPCs, EC2 instances, S3 storage, RDS, and networking configurations.
  • Ensure high availability, disaster recovery, and backup solutions for cloud environments.
  • Implement Infrastructure as Code (IaC) using tools like Terraform, AWS CloudFormation, or CDK.

• Data Solutions with Databricks:
  • Serve as a Databricks SME, guiding the architecture and implementation of ETL pipelines, data lakes, and machine learning workflows using Databricks on AWS.
  • Optimize Databricks clusters for performance, cost, and security.
  • Integrate Databricks with other AWS services like S3, Redshift, and Glue for seamless data processing.

• Leadership and Team Collaboration:
  • Lead a team of cloud engineers, providing mentorship and technical guidance.
  • Collaborate with cross-functional teams including data scientists, analysts, and developers to understand requirements and translate them into scalable cloud and data architectures.
  • Manage project timelines, ensuring timely delivery of cloud and data solutions.

• Security and Compliance:
  • Implement best practices for cloud security, including identity and access management (IAM), encryption, and VPC security.
  • Ensure compliance with industry standards (e.g., GDPR, HIPAA) and internal security policies.
  • Conduct regular security audits and optimize cloud configurations to reduce risk.

• Performance Optimization and Monitoring:
  • Set up monitoring, logging, and alerting using AWS CloudWatch, Datadog, or similar tools.
  • Analyze system performance and provide recommendations for improvement.
  • Troubleshoot complex issues related to cloud infrastructure, data pipelines, and Databricks clusters.

Minimum Skills and Qualifications:

• Technical Skills:
  • Strong expertise in AWS services (EC2, S3, Lambda, RDS, VPC, CloudFormation, IAM, etc.).
  • Proficiency with Databricks, including building and managing data pipelines, data lakes, and machine learning models.
  • Experience with Infrastructure as Code (IaC) tools like Terraform or AWS CloudFormation.
  • Knowledge of big data technologies like Apache Spark, Hadoop, or Delta Lake.
  • Proficient in programming and scripting languages (e.g., Python, SQL, Scala).
  • Experience with CI/CD pipelines for deploying cloud infrastructure.

• Soft Skills:
  • Strong leadership and project management skills.
  • Excellent problem-solving and analytical skills.
  • Ability to work collaboratively with diverse teams.
  • Strong communication skills, with the ability to explain technical concepts to non-technical stakeholders.

Certifications:

• Required:
  • AWS Certified Solutions Architect (Associate or Professional)
  • Databricks Certified Associate Developer for Apache Spark

• Preferred:
  • AWS Certified DevOps Engineer – Professional
  • AWS Certified Big Data – Specialty or AWS Certified Data Analytics – Specialty
  • Databricks Certified Professional Data Engineer