Big Data / PySpark Engineering Lead - Vice President
PLOT NO-1, S.NO. 77, India
INR 2000K-3000K (estimate) Senior-level Full Time
Tasks
- Benchmark performance against legacy systems
- Build CI CD pipelines for data jobs
- Collaborate with product managers on data availability
- Conduct code reviews and mentorship
- Conduct risk assessment and compliance support
- Design scalable data processing pipelines
- Develop data models and schema designs
- Implement data parity testing
- Implement data validation and encryption
- Integrate Big Data tools and frameworks
- Migrate legacy systems to data lakehouse
- Monitor system health and troubleshoot bottlenecks
- Optimize SQL queries
- Orchestrate phased migration cutover
- Perform schema evolution to Parquet Avro and Iceberg
- Re engineer ETL and ELT jobs with Spark
- Translate business requirements into technical specifications
- Tune distributed computing clusters
- Write high-performance Python code
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Hive | Apache Kafka | Apache Spark | Avro | Bitbucket | CD | CI | CI/CD | CSV | Collibra | Data Lakehouse | Data Lineage | Data Validation | ELT | ETL | Encryption | Git | HBase | Hadoop | Iceberg | Impala | Informatica | JSON | MongoDB | Parquet | Performance Benchmarking | Presto | PySpark | Python | SQL | Schema evolution | Shell Scripting | Spark SQL | Starburst | Trino | Unix | YARN
Education
N/A
Related jobs
-
A/B | A/B Testing | AutoGPT | B testing | Dimensionality ReductionExecutive-level Full TimeMumbai Goregaon, India3h ago
-
Machine Learning Engineer IV INR 4443K-7998KAPIs | Amazon Web Services | Automated testing | Cloud Native | Cloud platformBonuses | Disability insurance | Employee resource groups | Inclusive culture | Life insuranceSenior-level Full TimeRemote, India R5h ago
-
Senior-level Full TimePune City, India9h ago
-
Mid-level Full TimePune, Maharashtra, India9h ago
-
AM - Cloud Data Engineer INR 1500K-2500KAWS N/A | Apache Airflow | Apache Kafka | Apache Spark | Azure DataClient-facing experience | Consulting experience | Growth mindset cultureSenior-level Full TimeBangalore, Karnataka, India10h ago
-
AI Engineer – Reliability & Warranty Analytics INR 1800K-3000KBacktesting | BigQuery | CI/CD | Calibration | Cloud RunMid-level Full TimeChennai, Tamil Nadu, India10h ago
-
Associate III Business Intelligence-SUPPORT SERVICES-Data & Analytics - In House Engineering INR 800K-1800KBig Data | Big data concepts | C++ | Data Visualization | Data conceptsMid-level Full TimeBangalore, Karnataka, India11h ago
-
AI Engineer – Retail/CPG Domain INR 1800K-2500KAWS | Azure | Data Governance | Data Modeling | Data SecurityMid-level Full TimeGurugram, Haryana, India12h ago
-
Consultant - Python Developer with Gen AI and AI Foundry INR 2000K-4000KAWS | Autogen | Azure | Azure OpenAI | Azure OpenAI ServiceSenior-level Full TimeBangalore, Karnataka, India13h ago
-
Associate Principal Engineer (India Office) INR 1500K-2000KADLS | Azure Data | Azure Data Factory | Azure DevOps | CDCMid-level Full TimeBangalore, Karnataka, India13h ago
-
Senior-level Full TimePune, Maharashtra, India13h ago
-
Senior-level Full TimeChennai, Tamil Nadu, India13h ago
-
Applied AI ML Lead INR 2500K-4800KAI Agents | AWS Lambda | AWS S3 | AWS Step Functions | Agentic AISenior-level Full TimeHyderabad, Telangana, India13h ago
-
Software Engineer (Backend Developer) INR 1500K-3500KAWS | Active Directory | Agile | Algorithms | Data StructuresMid-level Full TimeBangalore, India Office (BANGALORE) R13h ago
-
Agentic AI | Agile | DAX | Data Governance | Data ModelingSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
Lead Software Engineer - Java/Python, LLM, AWS INR 3000K-5000KA/B | A/B Testing | AWS | Agile | Amazon AuroraSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
Platform Engineering Professional - Data Engineer INR 2200K-3500KAWS | Amazon Kinesis | Apache NiFi | Apache Spark | CI/CDMid-level Full TimeBengaluru, IN, 56010314h ago
-
AWS Data Engineer INR 1500K-6000KAWS | Automation | Cloud Architecture | Data Governance | Data IngestionExecutive-level Full TimeHyderabad, Telangana, India14h ago
-
Mid-level Full TimePune, Maharashtra, India15h ago
-
Senior-level Full TimeHyderabad, TG, India15h ago
-
Senior-level Full TimeNoida, Uttar Pradesh, India16h ago
-
Consultant - AI Engineer INR 2000K-4500KAutogen | CI/CD | Cloud infrastructure | CrewAI | Data PipelinesSenior-level Full TimeBangalore, Karnataka, India16h ago
-
Generative AI Engineer INR 2000K-4500KAmazon Web Services | Autogen | CI/CD | Cloud platform | CrewAISenior-level Full TimeHyderabad, Telangana, India16h ago
-
FastAPI | Flask | GIS | Git | MultiprocessingMid-level Full TimeNoida, Uttar Pradesh, India17h ago
-
Assistant Manager - WFM Performance Reporting & Analytics INR 770K-1200KDatabase Management | MIS Reporting | MS Access | MS Office | Motion studiesAnnual health checkup | Dental benefits | Group term life insurance | Inclusive health benefits | Learning and development programMid-level Full TimeGurugram, India17h ago