Big Data / PySpark Engineering Lead - Vice President
PLOT NO-1, S.NO. 77, India
INR 2000K-3000K (estimate) Senior-level Full Time
Tasks
- Benchmark performance against legacy systems
- Build CI CD pipelines for data jobs
- Collaborate with product managers on data availability
- Conduct code reviews and mentorship
- Conduct risk assessment and compliance support
- Design scalable data processing pipelines
- Develop data models and schema designs
- Implement data parity testing
- Implement data validation and encryption
- Integrate Big Data tools and frameworks
- Migrate legacy systems to data lakehouse
- Monitor system health and troubleshoot bottlenecks
- Optimize SQL queries
- Orchestrate phased migration cutover
- Perform schema evolution to Parquet Avro and Iceberg
- Re engineer ETL and ELT jobs with Spark
- Translate business requirements into technical specifications
- Tune distributed computing clusters
- Write high-performance Python code
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Hive | Apache Kafka | Apache Spark | Avro | Bitbucket | CD | CI | CI/CD | CSV | Collibra | Data Lakehouse | Data Lineage | Data Validation | ELT | ETL | Encryption | Git | HBase | Hadoop | Iceberg | Impala | Informatica | JSON | MongoDB | Parquet | Performance Benchmarking | Presto | PySpark | Python | SQL | Schema evolution | Shell Scripting | Spark SQL | Starburst | Trino | Unix | YARN
Education
N/A
Related jobs
-
Database Engineer INR 500K-1500KCI/CD | DB2 | Data Warehousing | Database performance | Database performance tuningMid-level Full TimePune, MH, India15h ago
-
Senior AI Engineer I INR 3264K-5876KArtificial Intelligence | BigQuery | Cloud Composer | Cloud platform | Data ProcessingSenior-level Full TimeBengaluru, KA, India15h ago
-
Principal Engineer, Data Analytics Engineering [ 8+ years ] INR 2500K-3584KAWS | Algorithms | Azure | Big Data | CI/CDSenior-level Full TimeBengaluru, KA, India15h ago
-
Staff AI/ML Engineer, YouTube Create INR 2500K-5000KAgentic framework | Algorithms | Artificial Intelligence | Audio Processing | Computer VisionSenior-level Full TimeBengaluru, Karnataka, India16h ago
-
Agent Development | Agent Development Kit | Angular | Backend Development | C++Mid-level Full TimeBengaluru, Karnataka, India16h ago
-
Assistant Director,Collections Analytics Lead INR 3000K-4000KAlteryx | Apache Spark | BigQuery | Capacity Planning | Cloud platformSenior-level Full TimeKolkata, West Bengal, India R16h ago
-
Data & Analytics Engineer - Manager INR 1500K-2000KAirflow | Azure Data | Azure Data Factory | Azure OpenAI | CI/CDMid-level Full TimeBangalore, Karnataka, India16h ago
-
Senior Engineer - Data Analyst INR 3000K-5012KAPI | Automation | Big Data | Big data analytics | CI/CDSenior-level Full TimeHadapsar, Pune, Maharashtra, IND, 41101316h ago
-
AWS | Advanced Excel | Agile Development | Apache Spark | AzureExecutive-level Full TimeKolkata, West Bengal, India R17h ago
-
Lead Software Engineer - Java and AI/ML INR 3500K-5000KAI Agents | Agile | Automated testing | CI/CD | Code modernizationSenior-level Full TimeMumbai, Maharashtra, India18h ago
-
AI Engineer I INR 2500K-5000KAWS | Azure | Azure OpenAI | Docker | FastAPIWork from office 5 days per weekSenior-level Full TimeMumbai, India18h ago
-
Mid-level Full TimePune, Maharashtra, India19h ago
-
Databricks Developer INR 1500K-2400KACID | Batch Processing | CI/CD | Cluster management | Data IngestionWork from office 5 days per weekMid-level Full TimeGurugram, Haryana, India19h ago
-
Associate-Digital Product Management INR 1685K-3300KAPI Integration | Anthropic | Data Governance | Data Ingestion | Data ValidationCareer growth opportunities | Holistic well-being support | Leadership development programs | Training opportunitiesMid-level Full TimeGurugram, HR, India19h ago
-
Associate-Digital Product Management INR 1685K-3300KAPIs | Anthropic | Data Quality | Data Visualization | Data integrationMid-level Full TimeGurugram, HR, India19h ago
-
Principal Data Scientist INR 3000K-4000KContainers | Continuous integration | Decoder Architecture | Deep learning | Design PatternsCar lease | Certification programs | Continuous learning programs | Corporate pension plan | Dental coverageSenior-level Full TimeBangalore, Karnataka, India20h ago
-
Lead Assistant Manager - Azure Data Engineer INR 1500K-2500KAgile | Amazon Redshift | Apache Airflow | Apache Hadoop | Apache HiveSenior-level Full TimeNoida, Uttar Pradesh, India20h ago
-
Executive - Data Governance INR 600K-840KCI/CD | Data Cleansing | Data Denormalization | Data Modeling | Data NormalizationExecutive-level Full TimeBangalore, Karnataka, India20h ago
-
Sr Data Engineer, Python + Spark (Data Federation skillset - Data Lakehouse - Eg: Starburst) - Chennai INR 1500K-2500KAWS EMR | AWS Glue | AWS S3 | Apache Hudi | Apache Iceberg401k retirement plan | Dental insurance | Medical insurance | Paid Holidays | Paid time offSenior-level Full TimeIndia20h ago
-
Mid-level Full TimeVadodara, GJ, India20h ago
-
Data Science - Associate Director INR 1500K-2700KCI/CD | Caching | Cloud Run | Cloud SQL | Cloud StorageMid-level Full TimeHyderabad, Telangana, India21h ago
-
Assistant Manager- Python Gen AI Engineer INR 1200K-2500KAI Agents | API Development | Agile | Containerization | Data PipelinesWork from client office 5 days per weekMid-level Full TimeBangalore, Karnataka, India21h ago
-
Cloud DevOps - Consultant INR 1500K-3000KAWS | Apache Airflow | Apache Kafka | Apache Spark | AzureMid-level Full TimeGurgaon, Haryana, India21h ago
-
Senior Consultant_Python Engineer - Senior INR 1500K-3465KAWS Glue | Agile | Amazon Web Services | Apache Airflow | CI/CDSenior-level Full TimeThane, IN, 40060422h ago
-
Senior Lead Software Engineer - Java/Python, AI INR 3500K-5500KApplication Programming | Application Programming Interfaces | Authentication | Authorization | Automated testingSenior-level Full TimeMumbai, Maharashtra, India22h ago