Senior Big Data & PySpark Engineer
INR 2040K-2800K (estimate) Senior-level Full Time
Tasks
- Collaborate with business stakeholders
- Contribute to data architecture
- Contribute to data governance
- Contribute to data modeling
- Design data pipelines
- Develop data pipelines
- Develop technical documentation
- Ensure data quality frameworks
- Implement data quality checks
- Implement data transformations
- Maintain data pipelines
- Mentor junior developers
- Optimize PySpark jobs
- Optimize Spark jobs
- Perform code reviews
- Translate requirements into technical specifications
- Troubleshoot data pipeline issues
- Use Flink for streaming processing
- Use Kafka for streaming data
- Use NiFi for data flow
Perks/Benefits
Skills/Tech-stack
AWS | AWS Glue | Amazon EMR | Apache Airflow | Apache Flink | Apache NiFi | Apache Spark | Azure | Azure Synapse | Data Governance | Data Modeling | Data Quality | Data Security | Data Warehousing | Databricks | Dataframe API | Docker | Git | Google Cloud | Google Dataproc | HDFS | Kafka | Kubernetes | NumPy | Object-Oriented | Object-oriented programming | Pandas | PySpark | Python | SQL | Spark SQL
Education
Roles
Big Data Engineer | Data Engineer | Engineer | Pyspark Engineer
Related jobs
-
Data Engineer INR 938K-1200KAzure Data | Azure Data Factory | Azure Databricks | Data Factory | ETLEmployee Assistance Program (EAP) | Flexible working environment | LinkedIn Learning | Volunteer time offMid-level Full TimeChennai, TN, India3h ago
-
Software Engineer + Gen AI (Fresher) INR 300K-540KAI Model Integration | AI model | API Integration | Algorithms | Data StructuresEntry-level Full TimePune, MH, India4h ago
-
IN-Associate_AI/ML Developer_MS Engg_Advisory_Kolkata INR 1500K-2000KApp Service | Azure App | Azure App Service | Azure Blob | Azure Blob StorageFlexibility programmes | Inclusive benefits | Mentorship | Wellbeing supportMid-level Full TimeKolkata DN 57, India11h ago
-
CI/CD | Couchbase | Design Patterns | Docker | GuiceSenior-level Full TimeGurugram 8 B, India11h ago
-
Sr. Azure Data Engineer INR 1500K-2000KAWS | Azure | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeINDIA - BENGALURU - HP, IN11h ago
-
IN-Associate 2_Azure Data Engineer_OC D&A_Advisory_Bhubaneswar INR 1500K-2000KAgile | Azure | Azure Data | Azure Data Factory | Azure Data LakeMid-level Full TimeBhubaneswar - Ihub, India11h ago
-
Mid-level Full TimeBangalore, KA, IN, 56214911h ago
-
Mid-level Full TimeIndia - Hyderabad11h ago
-
Senior-level Full TimeIndia - Hyderabad11h ago
-
Mid-level Full TimeIndia - Hyderabad11h ago
-
Mid-level Full TimeIndia - Hyderabad11h ago
-
Senior-level Full TimeIndia - Hyderabad11h ago
-
Mid-level Full TimeIndia - Hyderabad11h ago
-
Senior Engineer Data - MDM INR 1628K-2146KAgile | Amazon Web Services | Azure | Cloud Platforms | Cloud platformFlexible work environment | Internal mobility | Volunteering opportunities | Well-being | Work-life balanceSenior-level Full TimeKA Bangalore, India11h ago
-
Lead Engineer Data - MDM INR 1500K-2040KAPIs | Agile | Amazon Web Services | Azure | Cloud platformFlexible work environment | Internal mobility | Volunteering opportunities | Well-being | Work-life balanceSenior-level Full TimeKA Bangalore, India11h ago
-
Machine Learning Engineer, Chakra INR 2000K-4600KBenchmarking | Conversational AI | Data Pipelines | Deep learning | DockerMid-level Full TimeHybrid in Bangalore, India R15h ago
-
Technical Support Engineer INR 850K-1100KAmazon Web Services | Apache Hadoop | Apache Spark | Bash | Cloud platformFlexible paid time offMid-level Full TimeHyderabad, India17h ago
-
Senior-level Full TimeIndia18h ago
-
Software Engineer Lead (Python + Snowflake) INR 2040K-3100KAirflow | CI/CD | DBT | Dagster | Data ModelingSenior-level Full TimeIndia (Noida)18h ago
-
Principal Consultant-Manager, GenAI Engineer (Cloud: GCP) INR 2520K-3380KAgent Development | Agent Development Kit | Agent systems | Artificial Intelligence | Automated testingContinuous learning opportunities | MentorshipSenior-level Full TimeIndia-Hyderabad22h ago
-
Senior-level Full TimeIN-AP-Hyderabad22h ago
-
Senior-level Full TimeIN-MH-Pune22h ago
-
Lead-Affluent BIU & Analytics-AFFLUENT-Program & MIS INR 1960K-2600KA/B | A/B Testing | B testing | Clustering | Data GovernanceSenior-level Full TimeMumbai, Maharashtra, India22h ago
-
Developer, Data Analytics INR 1200K-1968KBigQuery | Cloud Composer | Cloud Storage | Cloud platform | Data WarehousingHybrid workSenior-level Full TimeIndia-Telangana-Hyderabad22h ago
-
Senior Data Engineer INR 1300K-2000KBigQuery | Cloud Composer | Cloud Storage | Cloud platform | Data WarehousingHybrid workSenior-level Full TimeIndia-Telangana-Hyderabad22h ago