Senior Data Engineer (Pyspark, AWS EMR, Airflow, SQL, Data Warehousing, ETL, Spark)
Tasks
- Align data exchange formats
- Collaborate in agile team
- Deploy data pipelines
- Design data pipelines
- Develop big data applications
- Ensure pipeline best practices
- Improve data reliability and quality
- Integrate AI based workflows
- Own data pipeline architecture
- Own pipelines and overall system
- Own technical implementations
- Test data pipelines
Perks/Benefits
Skills/Tech-stack
AWS Lambda | Amazon EMR | Amazon S3 | Amazon Web Services | Apache Airflow | Apache Iceberg | Apache Spark | CI/CD | Data Lake | GitHub | GitLab | Java | Lake Formation | PySpark | Python | SQL | Scala | Source Control | Terraform | Web Services
Education
N/A
Roles
Related jobs
-
Lead Software Engineer - Pyspark, AWS, Databricks INR 3000K-5000KAWS | Agile | Apache Kafka | Application Resiliency | Automated testingSenior-level Full TimeBengaluru, Karnataka, India3h ago
-
Sr Lead Software Engineer - Agentic AI INR 3500K-5000KA2A SDK | APIs | AWS Lambda | Amazon EKS | Amazon S3Senior-level Full TimeBengaluru, Karnataka, India3h ago
-
Data & Analytics Engineer - AM INR 1500K-2500KApache Spark | Data Modeling | Data Warehousing | Databricks | Delta LakeSenior-level Full TimeBangalore, Karnataka, India3h ago
-
Senior-level Full TimeIN-AP-Hyderabad3h ago
-
AWS DE- Gen AI INR 2000K-3300KAWS Glue | AWS Identity | AWS Identity and Access Management | AWS Identity and Access Management IAM | AWS LambdaSenior-level Full TimeIN-TN-Chennai3h ago
-
Data Engineer INR 1600K-2500KAmazon Web Services | Apache Spark | Cloud platform | Docker | Google CloudSenior-level Full TimeIN-AP-Hyderabad3h ago
-
Lead Software Engineer - Lead Data Architect INR 3000K-5000KAWS | CI/CD | Cassandra | Data Governance | Data LineageSenior-level Full TimeMumbai, Maharashtra, India4h ago
-
Lead Data Engineer - Strategic Data Provisioning Data Engineer INR 3845K-5000KAI for Data | AI/ML | AWS | Agile | Apache SparkSenior-level Full TimeBengaluru, Karnataka, India4h ago
-
Senior Product Development Engineer (Embedded System Firmware and Python, C programming) with 5 to 8 years of experience INR 660K-3000KARM | ASIC | Arc | Bitbucket | C#Entry-level Full TimeBengaluru, KA, India5h ago
-
Data Engineer INR 1500K-2200KAPI Integration | Data Modeling | Data Pipelines | Data Transformation | Data VisualizationMid-level Full TimeHyderabad, Telangana, India5h ago
-
Senior AI/ML Engineer – Generative AI INR 3700K-5000KA/B | A/B Testing | AI orchestration | API Integration | Agent systemsSenior-level Full TimeIndia6h ago
-
Mid-level Full TimeChennai, TN, India6h ago
-
Software Engineer II INR 1000K-2000KApache Spark | Azure Cosmos | Azure Cosmos DB | Azure Data | Azure Data FactoryMid-level Full TimeIndia, Telangana, Hyderabad7h ago
-
AI Assisted Development | APIs | AWS | AWS Lambda | AgileSenior-level Full TimeBengaluru, Karnataka, India7h ago
-
Assistant Manager- AI Engineer INR 1200K-2500KAmazon Web Services | Autogen | Azure | Cloud Computing | ContainerizationMid-level Full TimeBangalore, Karnataka, India7h ago
-
Senior AI Engineer I INR 3264K-5876KAPIs | Apache Airflow | BigQuery | Cloud platform | Data PipelinesSenior-level Full TimeBengaluru, KA, India8h ago
-
Sr. Specialist, Data Engineering INR 2771K-4000KAgile | Airflow | BigQuery | CI/CD | Cloud FunctionsRelocation assistance within countrySenior-level Full TimeMumbai, MH, IN8h ago
-
Lead Assistant Manager INR 2000K-3900KAWS IAM | Access Tokens | Amazon Web Services | Audit Logging | CI/CDSenior-level Full TimeIndia9h ago
-
AI Engineer - Agentic & Generative AI INR 1200K-2500KAgile Scrum | Anthropic | Autogen | Chunking | CrewAIMid-level Full TimePune, Maharashtra, IN9h ago
-
API Integration | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryMid-level Full TimePune, Maharashtra, India10h ago
-
Azure Blob | Azure Blob Storage | Azure Data | Azure Data Factory | Azure SQLMid-level Full TimePune, Maharashtra, India10h ago
-
Azure Blob | Azure Blob Storage | Azure Data | Azure Data Factory | Azure Data LakeMid-level Full TimePune, Maharashtra, India10h ago
-
IT engineer data lakehouse INR 1500K-2500K3NF | APIs | Audit trails | Azure | Azure DatabricksMobile and flexible working models | Sabbaticals | Training opportunitiesSenior-level Full TimeBengaluru, KA, India10h ago
-
Mid-level Full TimeBangalore, Karnataka, India11h ago
-
Lead Software Engineer INR 2500K-5000KAWS | Asynchronous programming | Authentication | Authorization | AutogenSenior-level Full TimeRemote, India R12h ago