Architect - Data Engineer
Tasks
- Communicate architecture decisions trade offs implementation approach
- Conduct architecture and design reviews
- Create architecture designs reference patterns technical specifications
- Define patterns for scalable fault tolerant maintainable Spark applications
- Design scalable resilient data processing solutions
- Drive continuous improvement in architecture practices engineering standards
- Drive improvements through partitioning query optimization resource management caching tuning
- Ensure SDLC governance standards compliance
- Establish observability monitoring reliability operational excellence standards
- Lead Apache Spark batch and streaming solutions
- Mentor engineers on distributed systems design Spark development data engineering best practices
- Participate in roadmap discussions
- Perform code reviews design reviews architecture assessments
- Promote CI CD automated testing infrastructure as code DevOps best practices
- Provide performance optimization and operational best practices
- Resolve complex distributed processing technical challenges
- Share knowledge on emerging data platform technologies architectural patterns
- Support implementation of scalable maintainable solutions
- Translate business requirements into technical solutions
Perks/Benefits
- Hybrid work schedule
- In person collaboration at assigned office location minimum two days per week
- Work-life flexibility
Skills/Tech-stack
AWS Athena | AWS ECS | AWS EKS | AWS EMR | AWS Glue | AWS Lambda | AWS S3 | AWS Step Functions | Amazon CloudWatch | Apache Iceberg | Apache Kafka | Apache Spark | Automated testing | BigQuery | CI/CD | Cloud Native | Cloud Native Architecture | Cloud Storage | Dataframes | Dataset APIs | DevOps | Distributed Systems | GCP Cloud | GCP Cloud Storage | GCP Dataproc | Google Dataflow | Hadoop | Hive | Infrastructure as Code | Java | Join optimization | Memory Management | Partitioning | PubSub | Python | Query Optimization | Resource Management | Scala | Shuffle Optimization | Spark SQL | Step Functions | Structured Streaming | Terraform | “as-code”
Education
N/A
Roles
Related jobs
-
Databricks and Splunk Engineer - Senior Associate INR 3264K-4500KAmazon Web Services | Analytics | Azure | Cloud Computing | Cyber ThreatEmployee networks | Flexible work/life support | Hybrid work schedule | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeBangalore, India R4h ago
-
Senior-level Full TimeBangalore North, India R12h ago
-
AI Engineer INR 450K-700KData Preprocessing | Deep learning | Language Processing | MLOps | Machine LearningCareer development | Hybrid work flexibilityExecutive-level Full TimeNoida, India R20h ago
-
Associate Principal Engineer, Dotnet AI Architect INR 1800K-4000K.Net Core | .Net Framework | AI-assisted coding | ARM | ASP.NET MVCMid-level Full TimeRemote, India R21h ago
-
Staff Software Engineer, Machine Learning INR 3000K-4580KC++ | Classification | Data Pipelines | Deep learning | EmbeddingsHealth and wellness programs | Time away for employeesSenior-level Full TimeBengaluru, KA, India R21h ago
-
Senior Software Engineer, Data Platform INR 2000K-4200KApache Pinot | Artifactory | Git | Go | JavaSenior-level Full TimeIndia (Remote) R21h ago
-
AI/ML Engineering Lead/(Gen AI,Python,LLM,ETL)Associate Director INR 2000K-4000KApache Airflow | Apache Beam | Auditability | Automated testing | BigQueryContinuous professional development | Flexible working | Inclusive diverse environment | Opportunities for growthMid-level Full TimePune, Maharashtra, India R22h ago
-
API | AWS SageMaker | Azure Machine Learning | CI/CD | Context engineeringAdvanced AI exposure | Collaborative environment | Continuous learning | Contract engagement | End-to-end ownershipSenior-level Contract Full TimeIndia R23h ago
-
Data Architect & Delivery Lead INR 3600K-5400KAgile | Airflow | Batch Processing | BigQuery | Canonical DataSenior-level Full TimeHyderabad, India R1d ago
-
Senior Agentic Engineer, AI Platform INR 3000K-5000KAPI Development | AWS | Agent Orchestration | Authentication | AutoscalingSenior-level Full TimeBengaluru R1d ago
-
SW Engineer (1 - 2 years of exp in .Net / Python, Full-stack development, Angular / React, GenAI) INR 1500K-2500K.NET | API Management | Agile | Angular | Azure DevOpsHybrid work schedule | Limited direction working environmentMid-level Full TimeBengaluru, India R1d ago
-
Senior-level Full TimeHyderabad, India R1d ago
-
Senior-level Full TimeIndia - Remote R1d ago
-
Senior AI Engineer INR 2475K-3465KAirflow | Amazon Web Services | Angular | Apache Beam | BigQueryCareer development | Healthcare benefits | Hybrid work | Learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Lead AI Engineer INR 2156K-3500KAWS | Agile | Apache Airflow | Apache Beam | Atlassian ConfluenceGuided career tracks | Healthcare benefits | Hybrid work setting | Online learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Cloud/Big Data Engineer INR 2771K-3900KAgile | Apache Hive | Apache Spark | Automation | BigQueryHybrid work schedule | Work-life flexibilitySenior-level Full TimeChennai, India R1d ago
-
Lead Cloud Data Engineer - Manager INR 2771K-4064KAWS | Airflow | Autosys | CICD | Cloud SecurityEmployee networks | Flexible work/life support | Hybrid work schedule | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeHyderabad, India R1d ago
-
Senior-level Full TimeBangalore, Karnātaka, India R1d ago
-
Senior-level Full TimeBangalore, Karnātaka, India R1d ago
-
Engineer, Java & Spark INR 2771K-4064KAgile | Amazon Web Services | Apache Spark | Automated testing | CI/CDHybrid workMid-level Full TimeChennai, India R1d ago
-
Principal Engineer - Agentic Engineering INR 2156K-3500KAgile | Airflow | Apache Beam | BigQuery | BigtableCareer growth | Healthcare benefits | Hybrid work environment | Online learning platform | Paid time offSenior-level Full TimeIND-Pune-Equifax Analytics-PEC, India R1d ago
-
Senior-level Full TimePune, India R1d ago
-
Senior Data Engineer INR 3600K-5000KAWS | Apache Airflow | Apache Spark | Azure | CI/CDFinancial wellness support | Flexible remote work on Fridays | Hybrid work schedule | Mental health support | Paid time offSenior-level Full TimeBengaluru, India R1d ago
-
Senior Software Engineer, MLOps/SRE INR 2475K-3500KAerospike | Amazon Web Services | Apache Airflow | Apache Spark | CI/CD401k pension | Disability insurance | Financial wellness support | Healthcare benefits | Life insuranceSenior-level Full TimeBengaluru, India R1d ago
-
Associate Principal Engineer, AI Architect INR 1800K-3300KAPI Integration | AWS | Agent Orchestration | Azure | Backend architectureMid-level Full TimeRemote, India R1d ago