Big Data / PySpark Engineering Lead - Vice President
PLOT NO-1, S.NO. 77, India
INR 2000K-3000K (estimate) Senior-level Full Time
Tasks
- Benchmark performance against legacy systems
- Build CI CD pipelines for data job deployment
- Collaborate with product managers for downstream analytics data availability
- Design scalable batch and real time data processing pipelines
- Develop data models and schema designs
- Ensure compliance with laws and regulations
- Implement data validation and encryption
- Implement schema evolution to Parquet Avro and Iceberg
- Integrate Spark Flink and Kafka into existing stack
- Migrate data and logic to data lakehouse
- Monitor system health and troubleshoot performance bottlenecks
- Optimize SQL queries and tune distributed computing clusters
- Orchestrate phased cutover with parallel run and shadow execution
- Perform data parity testing
- Provide technical mentorship and conduct code reviews
- Re engineer ETL and ELT jobs with Spark
- Translate business requirements into technical specifications
- Write high-performance Python code
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Hive | Apache Impala | Apache Kafka | Apache Spark | Automated testing | Avro | Bitbucket | CI/CD | Collibra | Data Lakehouse | Data Lineage | Data Validation | Data encryption | Database security | ELT | ETL | Git | HBase | Hadoop | Iceberg | Informatica | MongoDB | NoSQL | Parquet | Presto | PySpark | Python | SQL | Shell Scripting | Spark SQL | Starburst | Trino | Unix | YARN
Education
N/A
Roles
Big Data Engineer | Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Senior Embedded Software Development Engineer INR 2156K-3200KAgile | C# | CI/CD | CppUnit | EmulatorsAnnual bonus | Flexible hours | Health insurance | On-demand training | Paid volunteering leaveSenior-level Full TimeChennai, TN, IN, 6000975h ago
-
Data Engineer II INR 1500K-2000KApache Spark | Azure Data | Azure Data Explorer | Azure Data Factory | Azure SynapseMid-level Full TimeHyderabad, TS, IN7h ago
-
AWS | Agent systems | Agentic AI | Amazon SageMaker | Autonomous AgentsSenior-level Full TimeBengaluru Millenia, India13h ago
-
AWS SageMaker | Agent systems | Agentic AI | Autonomous Agents | Azure Machine LearningSenior-level Full TimeBengaluru Millenia, India13h ago
-
Agent systems | Amazon SageMaker | Autonomous Agents | Azure ML | Computer VisionFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India13h ago
-
Agent systems | Agentic AI | Amazon SageMaker | Autonomous Agents | Azure Machine LearningFlexibility programs | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India13h ago
-
Azure Databricks | CI/CD | Data Pipelines | Data integration | ETLSenior-level Full TimeGurugram 10 C, India13h ago
-
Azure Databricks | Data integration | ETL | Filtering | JoiningSenior-level Full TimeGurugram 10 C, India13h ago
-
IN_Associate_Python/Gen AI_AI in Delivery_Advisory_Bangalore INR 3000K-4000KAWS | Azure | CI/CD | Data Augmentation | Data PreprocessingFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India13h ago
-
IN_Associate_Python/Gen AI_AI in Delivery_Advisory_Bangalore INR 3000K-4000KAWS | Adversarial Networks | Autoregressive models | CI/CD | Cloud platformFlexible work arrangements | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India13h ago
-
IN_Associate_Python/Gen AI_AI in Delivery_Advisory_Bangalore INR 3000K-4000KAWS | Adversarial Networks | Autoregressive models | Azure | CI/CDFlexibility program | Inclusive benefits | Mentorship | Wellbeing supportSenior-level Full TimeBengaluru Millenia, India13h ago
-
Module Lead - GCP Data Engineer Job INR 1650K-3200KBigQuery | Bigtable | CI/CD | Cloud Build | Cloud NetworkingContinuous learning | Flexible work arrangements | Stable employmentSenior-level Full TimeHyderabad, TG, IN13h ago
-
Senior Principal Engineer - Data Engineering INR 3000K-4000KAWS | Apache Spark | Automation | Azure Data | Azure Data FactoryHybrid work environment | Inclusive culture | Professional development opportunities | Supportive leadersSenior-level Full TimePune - Panchshil, India R13h ago
-
AI machine learning | Azure | Cloud platform | Compliance | Data ArchitectureExecutive-level Full TimeKolkata DN 57, India13h ago
-
Senior Data Engineer INR 3000K-4000KApache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Machine LearningSenior-level Full TimeIT Centre Bengaluru GDTC, India13h ago
-
API Development | Async Programming | Autogen | CI/CD | CrewAISenior-level Full TimeBengaluru Millenia, India13h ago
-
Lead AI Designer– Agentic AI (L6) INR 2000K-3500KAPIs | AWS | Agent SDK | Asynchronous programming | AutogenSenior-level Full TimePune, India13h ago
-
Senior Software AI Engineer (Agentic Platform) INR 2500K-4500KAWS ECS | AWS Lambda | AWS SageMaker | Asynchronous programming | DockerSenior-level Full TimeIndia16h ago
-
Applied AI Engineer, Agentic Systems INR 2755K-5500K.NET | API Development | Agent Orchestration | Anthropic API | C SharpSenior-level Full TimeRemote - India R18h ago
-
Data Engineer INR 1200K-1496KApache Airflow | Apache Spark | BigQuery | CI/CD | Cloud BuildEmployee assistance program | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India19h ago
-
Data Integration Engineer INR 1200K-2000KCycle management | Data Modeling | Data Quality | Data Reconciliation | Data ValidationSenior-level Full TimeHyderabad, Telangana, US21h ago
-
Staff Software Engineer - Semantic Layer INR 3000K-4000KAPI Design | Amazon Redshift | Backwards Compatibility | BigQuery | CompilationSenior-level Full TimeIndia - Remote R21h ago
-
Senior Associate - Data INR 1500K-2000KAzure Data | Azure Data Factory | Azure DevOps | Azure SQL | CI/CDMid-level Full TimePune21h ago
-
Lead Associate - Data Architect INR 1500K-2000KAmazon Web Services | Automated data pipelines | Azure | Business Intelligence | Data GovernanceMid-level Full TimePune22h ago
-
Sr. DevOps Engineer INR 2500K-3500KAIOps | AKS | ARM | Azure Container | Azure Container RegistryRotational on-call supportSenior-level Full TimeHyderabad, India23h ago