Big Data / PySpark Engineering Lead - Vice President
PLOT NO-1, S.NO. 77, India
INR 2000K-3000K (estimate) Senior-level Full Time
Tasks
- Benchmark performance against legacy systems
- Build CI CD pipelines for data job deployment
- Collaborate with product managers for downstream analytics data availability
- Design scalable batch and real time data processing pipelines
- Develop data models and schema designs
- Ensure compliance with laws and regulations
- Implement data validation and encryption
- Implement schema evolution to Parquet Avro and Iceberg
- Integrate Spark Flink and Kafka into existing stack
- Migrate data and logic to data lakehouse
- Monitor system health and troubleshoot performance bottlenecks
- Optimize SQL queries and tune distributed computing clusters
- Orchestrate phased cutover with parallel run and shadow execution
- Perform data parity testing
- Provide technical mentorship and conduct code reviews
- Re engineer ETL and ELT jobs with Spark
- Translate business requirements into technical specifications
- Write high-performance Python code
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Hive | Apache Impala | Apache Kafka | Apache Spark | Automated testing | Avro | Bitbucket | CI/CD | Collibra | Data Lakehouse | Data Lineage | Data Validation | Data encryption | Database security | ELT | ETL | Git | HBase | Hadoop | Iceberg | Informatica | MongoDB | NoSQL | Parquet | Presto | PySpark | Python | SQL | Shell Scripting | Spark SQL | Starburst | Trino | Unix | YARN
Education
N/A
Roles
Big Data Engineer | Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Python Robot QA INR 1524K-2541KAgile | Azure DevOps | Backup and Recovery | Case management | Continuous integrationHybrid workSenior-level Full TimeIN-AP-Hyderabad6h ago
-
Senior-level Full TimeIN-KA-Bangalore6h ago
-
ML and GenAI INR 1000K-1000KAPIs | Algorithms | Data Structures | Event Driven | Event-driven architectureMid-level Full TimeGurgaon, Haryana, India6h ago
-
RAG + Agentic AI Engineer (India) (Remote) INR 2500K-3500KAccess Control | Agent Orchestration | Asynchronous programming | Auditability | BM25Growth opportunities | Ownership and autonomy | Remote workMid-level Full TimeMaharashtra, Pune, India R6h ago
-
Staff Engineer, Data Engineering INR 2400K-3100KAPIs | Airflow | Apache Kafka | Apache Spark | AzureSenior-level Full TimeBengaluru, KA, India6h ago
-
DevOps Lead INR 3000K-5000KAmazon Web Services | Azure DevOps | CI/CD | Cloud Security | CloudFormationCareer growth opportunities | Flexible work environment | Hybrid work environmentSenior-level Full TimeKARNATAKA, Bengaluru, India7h ago
-
Mid-level Full TimeHyderabad, TS, IN; Bengaluru, KA, IN7h ago
-
Sr. Staff Engineer (Java Fullstack or Python, AI) INR 2475K-3500KAWS | Angular | Anthropic | Azure OpenAI | Bias MitigationFamily well-being benefits | Health benefits | Paid time off | Work-life balanceSenior-level Full TimeHyderabad, India7h ago
-
Senior Generative AI Engineer (India) (Remote) INR 2000K-4800KAI orchestration | AI/ML stack | API Development | API Gateway | AWS AIAgile environment | Professional growth | Remote workSenior-level Full TimeMaharashtra, Pune, India R8h ago
-
Business Intelligence Engineer INR 2000K-2900KAgile | Bitbucket | DAX | Data Governance | Data WarehousingFlexible hybrid work model | Health insurance | Life insurance | Paid time off | Pension/retirement benefitsMid-level Full TimeHyderabad, India R8h ago
-
Data Engineer INR 1000K-2100KApache Airflow | Apache Spark | Azure | CI/CD | DatabricksEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India8h ago
-
Mid-level Full TimeMaharashtra, Pune, India8h ago
-
AI Data Scientist INR 1800K-3000KAgent systems | Agentic Systems | Calculus | Data Preprocessing | Diffusion ModelsMid-level Full TimeChennai, Tamil Nadu, India9h ago
-
Senior Software Engineer (Microsoft .NET/Azure cloud) INR 2829K-4000K.NET | API Management | APIM | Angular | App ServicesHealth insurance | Paid time off | Wellbeing benefits | Work-life balanceSenior-level Full TimeHyderabad, India9h ago
-
Lead Data Engineer - Data Modeler INR 1500K-2400KAmazon EC2 | Amazon EMR | Amazon RDS | Amazon Redshift | Amazon Web ServicesSenior-level Full TimePune, Maharashtra, India9h ago
-
Consultant - AI Engineer INR 2000K-3500KAutogen | CI/CD | Claude | Cloud infrastructure | ContainerizationSenior-level Full TimeBangalore, Karnataka, India10h ago
-
MLOps Engineer (Generative AI) INR 1500K-2100KAgentic pipelines | Agile | Amazon Web Services | Azure | Azure Machine LearningEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India10h ago
-
Machine Learning (Generative AI) INR 1500K-2500KAgile | Azure Machine Learning | CI/CD | Computer Vision | ConfluenceEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India10h ago
-
Senior-level Full TimeMumbai, Maharashtra, India10h ago
-
Mid-level Full TimeMumbai, Maharashtra, India10h ago
-
Generative AI Engineer INR 2000K-3500KAI Studio | AWS SageMaker | Agents | Amazon Bedrock | Anthropic APICollaborative team environment | Leadership development | Training programsMid-level Full TimeMumbai, Maharashtra, India10h ago
-
Staff GenAI Engineer INR 2000K-3500KAWS | Agentic AI | Azure | CI/CD | Distributed SystemsCollaborative team environment | Leadership development programs | Training sessionsSenior-level Full TimeMumbai, MH, India10h ago
-
API Testing | Apache Airflow | Azure | CI/CD | DatabricksEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, India11h ago
-
Tech Manager- GTM Applied AI & Analytics INR 2000K-3380KAirflow | Apache Spark | Databricks | Fine Tuning | LangchainFlexible work schedule | Hybrid work scheduleSenior-level Full TimeBengaluru, KA, India11h ago
-
Mid-level Full TimeGurugram, India11h ago