Data Engineers (Big Data Hadoop, Scala, Spark, Ozone/Iceberg/Airflow)
Tasks
- Build CI CD for data pipelines
- Build data pipelines
- Implement table formats for data lakes
- Manage data lake storage and compaction
- Monitor and schedule data workflows
- Optimize Iceberg tables
- Orchestrate workflows with Apache Airflow
- Process data with Apache Spark
- Run distributed SQL queries
- Tune Spark performance
- Write complex SQL queries
Perks/Benefits
- N/A
Skills/Tech-stack
ACID transactions | Apache Airflow | Apache Hive | Apache Iceberg | Apache Ozone | Apache Spark | Avro | Broadcast Join | CI/CD | Catalyst Optimizer | Compaction | DAG | Data Skew | Dataframes | Datasets | Docker | Dynamic Task Mapping | Execution Plans | Git | HDFS | Lazy evaluation | Memory Management | Merge Join | ORC | Parquet | Python | Query Execution | Query execution plans | SQL | Scala | Scalatest | Schema evolution | Shuffle | Snapshot Isolation | Sort Merge Join | Spark SQL | Structured Streaming | Task Groups | Task Mapping | TaskFlow API | Time Travel | Tungsten | Unit Testing | Window Functions | XCOM | YARN
Education
N/A
Roles
Related jobs
-
Camera Systems Software Engineer - India INR 1200K-2500KBandwidth Management | Bayer | Buffering | C# | C++Mid-level Full TimeBengaluru, India10h ago
-
Data Engineer - People Data, Insights & Analytics INR 1500K-2000KAWS | Apache Spark | Azure | CI/CD | Data ModelingMid-level Full TimePune, Maharashtra, India10h ago
-
Senior-level Full Timecoimbatore, India16h ago
-
Python Engineer (AI/ML) INR 2500K-3500KCPU | Chroma | Cost Optimization | Data Preprocessing | DockerMid-level Full TimePune, MH, India17h ago
-
Data Analytics AI Specialist INR 15K-20KClassification | Clustering | Data Visualization | Data cleaning | Machine LearningFast-paced startup environment | Growth opportunities | High ownershipEntry-level Full TimeGurugram, HR, IN18h ago
-
Senior-level Full TimePune City, India18h ago
-
Senior-level Full TimeIN-KA-Bangalore19h ago
-
Gen AI Developer INR 2000K-4500KData Engineering | Distributed Systems | GenAI | Language Models | Large Language ModelsSenior-level Full TimeIN-TN-Chennai19h ago
-
Senior-level Full TimeIN-TN-Chennai19h ago
-
Senior-level Full TimeIN-TN-Chennai19h ago
-
Senior-level Full TimeIN-TN-Chennai19h ago
-
Software Engineer - Azure Data engineer INR 3000K-3500KAccess Control | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageCollaborative work environment | In-office 5 days | Professional development | Recognition and rewards | Training opportunitiesSenior-level Full TimeBangalore - Karnataka, India - Kalyani …20h ago
-
Lead Data Engineer - Data Archival - Bangalore INR 1500K-2500KApache Spark | Data Warehousing | ETL | Hadoop | JavaSenior-level Full TimeIndia20h ago
-
Senior-level Full TimePune, Maharashtra, India21h ago
-
Azure Infra AI Operations & Platform Engineer INR 2000K-4500KAI Foundry | AKS | API Integration | Azure | Azure AIMid-level Full Timebangalore, India21h ago
-
Quantum AI Research Engineer INR 180K-240KAI Agents | Artificial Intelligence | Data Analysis | Deep learning | Generative AIEntry-level Full Timebengaluru, India21h ago
-
Data Engineer — Recommendation Engine INR 1500K-2200KAWS Glue | Airflow | Amazon ECS | Amazon S3 | Cache invalidationMid-level Full TimeHaryana, Gurugram, India21h ago
-
Machine Learning Engineer (Remote) INR 2500K-4500KAutomated testing | CI/CD | Git | Machine Learning | NumPyRemote work | Scrum team collaborationSenior-level Full TimeMaharashtra, Pune, India R22h ago
-
Senior Data Engineer INR 3000K-4000KAWS | Amazon Aurora | Apache Spark | Azure | Batch ProcessingAccreditations | Bike to work | Educational assistance | Employee assistance program | Employee discountSenior-level Full TimeMumbai, MH, India22h ago
-
AI Engineer (Manager level) INR 1800K-2500KAI Services | Apache Spark | Azure AI | Azure AI Services | Azure CIMid-level Full TimeHyderabad, India23h ago
-
Lead Software Engineer - Machine Learning INR 3000K-5000KCross-validation | Data Pipelines | Distributed Systems | Low Latency | MLOpsSenior-level Full TimeChennai, India23h ago
-
Mid-level Full TimeBengaluru, Karnataka, India1d ago
-
Snowflake Lead - Manager INR 1500K-2500KAirflow | Anthropic Claude | Azure Data | Azure Data Factory | Azure DevOpsSenior-level Full TimeBangalore, Karnataka, India1d ago
-
Mid-level Full TimeHyderabad, TS, India1d ago
-
AES - Generative AI INR 1500K-2700KAI Agents | AI Evaluation | AI orchestration | API Gateway | AWSMid-level Full TimeBangalore, Karnataka, India1d ago