Senior Software Engineer - Big Data / Streaming Platforms (Spark / Kafka / NiFi)
Tasks
- Analyze requirements to design test scenarios
- Automate tests with SQL PySpark and Python
- Build test automation frameworks for data validation
- Develop test plans and test cases
- Execute functional regression and integration tests
- Execute performance load and soak testing
- Integrate tests into CI/CD pipelines
- Manage defects in JIRA
- Perform production validation and post deployment checks
- Support incident analysis with data validation and replay
- Test Apache NiFi ingestion routing and error handling
- Test Kafka streaming pipelines
- Validate JSON payloads and data flows
- Validate observability metrics like lag throughput latency and error rates
- Verify event replay with offsets and checkpoints
Perks/Benefits
- N/A
Skills/Tech-stack
Apache NiFi | Apache Ozone | Apache Spark | CI/CD | Ceph | Data Reconciliation | Data Validation | Distributed Systems | Git | JSON | Jira | Kafka | Linux | Load Testing | Object storage | Observability | Performance Testing | PySpark | Python | SQL | Shell Scripting | Soak Testing | Test automation
Education
Roles
Engineer | QA Engineer | SDET | Senior QA | Senior QA Engineer | Senior Software Engineer | Software Engineer
Related jobs
-
AI Engineer – Python & Snowflake INR 1800K-2700KAPI Development | AWS | Airflow | Azure | Data ModelingMid-level Full TimeHyderabad, Hyderabad, IN3h ago
-
Senior AI Engineer (Full-Stack) (Remote) INR 2500K-5000KAWS | Active Directory | Anthropic | ArcGIS REST | ArcGIS REST APIClient facing ownership | Remote workSenior-level Full TimeMaharashtra, Pune, India R3h ago
-
Senior Database Engineer - #3180 (Remote) INR 1500K-2700KAWS CDK | AWS IAM | AWS KMS | Amazon Aurora | Amazon CloudFormationSenior-level Full TimeChennai, India R3h ago
-
Lead AI Software Engineer INR 2000K-4500KAI Agents | AWS | Azure | CI/CD | Cloud platformCareer growth | Career growth in technology domain | Continuous learningSenior-level Full TimePune City, India4h ago
-
Senior-level Full TimeHyderabad, India4h ago
-
Senior-level Full TimeIN-HR-Gurgaon5h ago
-
Senior-level Full TimeIN-KA-Bangalore5h ago
-
DevOps Track Sr.Engineer INR 2229K-4000KAPI Gateway | APM | AWS CloudWatch | AWS Systems Manager | AWS systemsSenior-level Full TimeIndia7h ago
-
Database Track Sr.Engineer INR 2000K-3000KData Architecture | Database Schemas | Database performance | Database performance tuning | Disaster RecoverySenior-level Full TimeIndia7h ago
-
Big Data Engineer INR 1500K-2400KApache Spark | Azure Data | Azure Data Factory | Data Factory | Data IngestionSenior-level Full TimeIndia7h ago
-
Gen AI Developer Specialist INR 1500K-6000KAI Services | Anthropic | Azure AI | Azure AI Services | CI/CDExecutive-level Full TimeIndia7h ago
-
Gen AI Developer Specialist INR 1500K-6000KAI Search | AI Services | Anthropic | Azure AI | Azure AI ServicesExecutive-level Full TimeIndia7h ago
-
Gen AI Developer Specialist INR 1500K-6000KAI Services | Anthropic | Azure AI | Azure AI Services | CI/CDExecutive-level Full TimeIndia7h ago
-
Gen AI Developer Specialist INR 1500K-6000KAI Services | Anthropic | Azure AI | Azure AI Services | CI/CDExecutive-level Full TimeIndia7h ago
-
Lead-Affluent BIU & Analytics-AFFLUENT-Program & MIS INR 2000K-3500KA/B | A/B Testing | B testing | Clustering | Data WarehousingSenior-level Full TimeMumbai, Maharashtra, India7h ago
-
Big Data Lead INR 1500K-2000KAzure Data | Azure Data Lake | Azure Data Lake Storage | CI/CD | Cloud StorageSenior-level Full TimeIndia7h ago
-
Amaze - Cloud Data Architect INR 1500K-2500KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DatabricksSenior-level Full TimeIndia7h ago
-
Big Data Lead INR 2000K-3000KApache Airflow | Apache Beam | Apache Spark | BigQuery | Cloud ComposerSenior-level Full TimeIndia7h ago
-
Big Data Engineer INR 1500K-2000KApache Airflow | Apache Beam | Apache Spark | BigQuery | Cloud ComposerMid-level Full TimeIndia7h ago
-
Big Data Lead INR 1500K-2500KApp Service | Application Monitoring | Azure | Azure App | Azure App ServiceSenior-level Full TimeIndia7h ago
-
Deputy Manager - Fraud Analytics & Monitoring (Detection) INR 950K-1500KAnomaly Detection | DPSC | Data Visualization | Decisioning | Early WarningAnnual health checkup | Dental benefits | Gender neutral policy | Group term life insurance | Learning and development programMid-level Full TimeGurugram, India10h ago
-
Analyst-Data Analytics INR 750K-2000KCloud Computing | Data Mining | Data Processing | Data Validation | ExcelCareer growth opportunities | Comprehensive well-being support | Learning opportunitiesMid-level Full TimeGurugram, HR, India10h ago
-
AI/ML Engineer INR 2500K-3500KAPI Development | BigQuery | CI/CD | Classification | Cloud FunctionsInnovation culture | Personal growth | Professional growthSenior-level Full TimeBengaluru , Pune10h ago
-
Microsoft Fabric Data Engineer INR 1500K-2443KAPI Integration | Azure Data | Azure Data Factory | Azure Data Lake | Azure SQLMid-level Full TimeBengaluru, India10h ago
-
Senior Azure Data Engineer INR 1500K-2500KAzure | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimePune, Maharashtra, India11h ago