PySpark Data Engineer with Cloudera and Cloud Expertise
Bengaluru - BCIT, India
INR 2400K-3500K (estimate) Senior-level Full Time
Tasks
- Automate workflow orchestration
- Design data pipelines
- Develop PySpark transformations
- Implement data quality checks
- Maintain data governance documentation
- Manage data ingestion
- Monitor and troubleshoot pipelines
- Optimize spark job performance
- Support security and compliance
Perks/Benefits
Skills/Tech-stack
AWS EMR | Apache Airflow | Apache Atlas | Apache Oozie | Apache Spark | Azure HDInsight | Bash | CI/CD | Cloud Dataproc | Cloudera Data Platform | Collibra | Data platform | GitLab CI | Google Cloud | Google Cloud Dataproc | HBase | HDFS | Hadoop | Hive | Impala | Jenkins | Kafka | Linux | PySpark | Python | SQL
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Senior Software Engineer, AI and Infrastructure, CMCS INR 2500K-5000KAPIs | As-a-Service | C++ | CPU | Data ProcessingSenior-level Full TimeHyderabad, Telangana, India3h ago
-
Applied Artificial Intelligence & Machine Learning- Markets Operations - Vice President INR 3000K-3000KAWS | Agile | Azure | Containerization | Data PreprocessingExecutive-level Full TimeBengaluru, Karnataka, India5h ago
-
Lead Software Engineer - Java, Streaming, AWS INR 2000K-3500KAWS | Agile | Amazon EKS | Amazon EMR | Amazon KinesisSenior-level Full TimeHyderabad, Telangana, India6h ago
-
Software Engineer III - AWS, PySpark, ETL INR 2050K-3300KAWS | AWS EMR | Agile | Apache Spark | CI/CDSenior-level Full TimeHyderabad, Telangana, India7h ago
-
Sr. Data Engineer USD 127K-164KAgentic AI | DBT | Data Governance | Data Quality | Data SecurityComprehensive benefits | Extensive training initiatives | Group insurance | Growth opportunities | Hospitalization insuranceSenior-level Full TimeBangalore, FL, US, 5600768h ago
-
Value Reporting Data Engineer INR 1431K-2000KData Governance | Data Lakehouse | Data pipeline | Databricks | DataverseMid-level Full TimeKochi, Kerala, India8h ago
-
Python , web scrapping data conversion VBA INR 1500K-2156KAPI Integration | Alerting | BeautifulSoup | Data Automation | Data EngineeringHybrid working | UK time zone collaborationSenior-level Full TimeGurgaon, HR, India9h ago
-
Manager - Azure Data Engineering INR 1500K-1700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsMid-level Full TimeBangalore, Karnataka, India9h ago
-
Assistant Manager - Azure Data Engineering INR 1500K-1700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsMid-level Full TimeBangalore, Karnataka, India9h ago
-
Assistant Manager - Azure Data Engineering INR 1500K-1700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsMid-level Full TimeBangalore, Karnataka, India9h ago
-
Senior - Azure Data Engineering INR 2500K-4000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full TimeBangalore, Karnataka, India9h ago
-
Senior - Azure Data Engineering INR 2500K-4000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full TimeBangalore, Karnataka, India9h ago
-
Senior - Azure Data Engineering INR 2500K-4000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full TimeBangalore, Karnataka, India9h ago
-
Senior - Azure Data Engineering INR 2500K-4000KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsSenior-level Full TimeBangalore, Karnataka, India9h ago
-
Senior Data Engineer INR 2520K-3500KAI Platform | Apache Airflow | Apache Spark | CI/CD | Cloud StorageBackground checks during recruitment | Creche allowance | Cultural events | Flexible work model | Lab daysSenior-level Full TimeBangalore9h ago
-
Assistant Manager - Azure Data Engineering INR 1500K-1700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DevOpsMid-level Full TimeBangalore, Karnataka, India9h ago
-
IN_Senior Associate_Agentic AI Developer_One cloud_Advisory_Pune INR 2755K-4225KAWS Bedrock | CI/CD | Deep learning | Faiss | FastAPIFlexible work arrangements | Mentorship | Wellbeing supportSenior-level Full TimePune, India14h ago
-
Senior-level Full TimeBangalore, KA, IN14h ago
-
Tech Lead - Data Bricks Job INR 2040K-3500KAgile | Azure | Azure Data | Azure Data Factory | Azure DatabricksContinuous learning | Flexible work arrangementsSenior-level Full TimeHyderabad, TG, IN14h ago
-
Senior AI Engineer INR 2500K-4800KAI auditability | AI guardrails | Accelerate | Access Controls | AdaptersSenior-level Full TimeBengaluru, Karnataka, India14h ago
-
Senior Embedded Software Engineer INR 1326K-2750KC# | C++ | CI/CD | Device Driver Development | Device driverContinuous learning opportunities | Mentorship | Professional growth | Work-life balanceSenior-level Full TimeIndia, Bangalore, Nova14h ago
-
Lead/Senior Software Engineer – User Foundations Platform INR 2500K-3500KAWS | Azure | CI/CD | Databases | Distributed SystemsSenior-level Full TimeIND - Bangalore - Ecoworld 6B …14h ago
-
AI Engineer (GDC)-65851 INR 60K-180KAWS | Agentic AI | CI/CD | Data Engineering | FastAPIFlexible work arrangements | Health and wellbeing benefits | Work-life balanceEntry-level Full Time(DEAI DS) IN Bengaluru - Residency …14h ago
-
V&V Engineer - Med Tech / Embedded System INR 480K-900KC plus plus | C# | CAN | CI/CD | Defect ManagementMid-level Full Timecoimbatore, India21h ago
-
Technology Product Owner I INR 2500K-3500KAWS Step Functions | Amazon DynamoDB | Amazon S3 | Amazon SQS | Amazon Web ServicesDevelopment opportunities | Reward and recognition | Volunteering opportunities | Wellness | Work-life balanceMid-level Full TimeMumbai, India21h ago