Senior Data Engineer
Tasks
- Build and operate Kafka and Spark Structured Streaming pipelines
- Build ingestion and routing flows with Apache NiFi
- Collaborate on architecture and code reviews
- Design and develop Spark applications in Scala and PySpark
- Document architecture and engineering best practices
- Ensure data quality consistency and auditability
- Handle late data and event replay with Kafka offsets
- Implement ETL ELT pipelines with low latency fault tolerance
- Implement stateful streaming with windowing watermarking checkpointing
- Integrate Spark with Apache Ozone and Ceph
- Monitor production systems and perform incident analysis
- Optimize Spark jobs with partitioning memory tuning shuffle optimization
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Kafka | Apache NiFi | Apache Ozone | Apache Spark | CI/CD | Ceph | Checkpointing | Distributed Systems | ELT | ETL | Git | Kafka Offsets | Linux | Logging | Memory tuning | Monitoring | Object storage | Partitioning | PySpark | Python | SQL | Scala | Shell Scripting | Shuffle Optimization | Spark Structured Streaming | Stateful Streaming | Structured Streaming | Watermarking | Windowing
Education
Roles
Related jobs
-
Software Engineer (Data Engineering & Building APIs) INR 700K-1500KAWS | Algorithms | CI/CD | Data Structures | DevOpsHybrid work flexibilityMid-level Full TimeGurgaon - Cyber Park, India2h ago
-
Senior-level Full TimeBangalore8h ago
-
Senior-level ContractHyderabad, India9h ago
-
Mid-level Full Timekerala, India9h ago
-
Senior-level Full TimeNoida,UP,India R9h ago
-
Lead Software Engineer - Java, Spring boot, AWS Cloud & AI/ML (REST APIs, System Design, SDLC) INR 3000K-5000KAPI Gateway | AWS Glue | AWS Lambda | AWS Step Functions | AgileSenior-level Full TimeBengaluru, Karnataka, India10h ago
-
Senior-level Full TimeIN-MH-Pune10h ago
-
Data Engineer INR 1500K-2500KAWS | AWS Glue | AWS Lake Formation | AWS Step Functions | Amazon AthenaSenior-level Full TimeIN-MH-Pune10h ago
-
Snowflake Developer INR 1500K-2500KApache Airflow | DBT | Data Sharing | Data Warehousing | Dynamic TablesSenior-level Full TimeIN-TN-Chennai10h ago
-
Associate Engineer - AI/ML INR 2500K-4000KDistributed Systems | Edge AI | GPU | LLM Agents | LangchainAccess to cutting edge technologies and tools | Collaborative work environment | Professional growth and development | Training and development opportunitiesMid-level Full TimeBangalore - Karnataka, India - Kalyani …10h ago
-
Technologist, Data Analytics Engineering INR 3000K-4500KAI orchestration | API Management | Artificial Intelligence | Automation | CI/CDSenior-level Full TimeBengaluru, KA, India11h ago
-
GenAI Specialist INR 2000K-5000KAI Studio | Agent systems | Agentic AI | Azure AI | Azure AI StudioLeadership development | Training and development programsSenior-level Full TimeBengaluru, India11h ago
-
Senior Expert D&T MLOps Engineer INR 3000K-4000KDBT | Data Modeling | Data Vault | Databricks | ETLSenior-level Full TimeHyderabad, India11h ago
-
AI Customer Engineer, Conglomerates, Google Cloud INR 3000K-5000KAI Agent Development Kit | Agent Development Kit | Agentic AI | As-a-Service | Audit LoggingSenior-level Full TimeGurugram, Haryana, India; Mumbai, Maharashtra, India11h ago
-
Mid-level Full TimeIndia13h ago
-
GenAI Engineer INR 1500K-3000KAmazon Web Services | Apache Spark | Azure | Chunking | Data StructuresMid-level Full TimeIndia13h ago
-
Embedded Software Engineer - India INR 1600K-3000KAssembly | Auto-code | Auto-code generation | C# | C++Mid-level Full TimeAiroli, IN-MH, 400708, IND14h ago
-
Anthropic SDK | Embeddings | Fine Tuning | Git | GitHub ActionsCollaborative values led team | Exposure to global teams and customers | Generous holiday allowance | Hybrid work environment | Individual career developmentEntry-level Apprenticeship Full TimeWhitefield, Bangalore, India14h ago
-
ML-Ops INR 3500K-5000KAsyncio | Benchmarking | CI/CD | CProfile | ConcurrencyLeadership development programs | Training sessionsSenior-level Full TimeMumbai, Maharashtra, India15h ago
-
Staff GenAI Engineer INR 2500K-5000KAWS | Agentic Systems | Azure | CI/CD | Distributed SystemsCollaborative team environment | Leadership development programs | Training sessionsSenior-level Full TimeMumbai, MH, India15h ago
-
Mid-level Full TimeMumbai, Maharashtra, India15h ago
-
Senior-level Full TimeMumbai, Maharashtra, India15h ago
-
Generative AI Engineer INR 1800K-3300KAI Studio | AWS SageMaker | Amazon Bedrock | Anthropic API | Azure AIGlobal consulting projects | Leadership trainingMid-level Full TimeMumbai, Maharashtra, India15h ago
-
Software Development Senior Associate INR 2000K-4500KAmazon Web Services | Artificial Intelligence | CI/CD | Data Analysis | DockerCareer growth opportunities | Certification support | Collaborative work environment | MentorshipSenior-level Full TimeChennai, TN, IN16h ago
-
AI Platform Engineer / Senior Consultant Specialist INR 3000K-5000KAPI Development | Ansible | Batch inference | CI/CD | DockerSenior-level Full TimeHyderabad, Telangana, India R16h ago