Big Data Lead
Tasks
- Build ELT pipelines
- Build ETL pipelines
- Collaborate on metadata and access control
- Configure metrics and alerts
- Design data models
- Document data sources and transformations
- Document runbooks
- Implement ABAC
- Implement CDC
- Implement RBAC
- Implement automated testing
- Implement data lineage
- Implement data quality checks
- Implement micro batch processing
- Implement security and privacy controls
- Implement streaming data pipelines
- Lead complex integrations
- Lead cross-team projects
- Lead data migrations
- Mentor engineers
- Optimize storage and compute
- Set up observability
- Write Python code
- Write SQL queries
Perks/Benefits
- N/A
Skills/Tech-stack
ABAC | Amazon Kinesis | Apache Airflow | Azure Data | Azure Data Factory | Azure Databricks | Azure Event | Azure Event Hubs | Bicep | CI/CD | Change Data Capture | DBT | Data Capture | Data Contracts | Data Factory | Data Lineage | Data Quality | Data Vault | Delta Lake | Dimensional Modeling | Event Hubs | Kafka | Observability | Parquet | Python | RBAC | SQL | Scala | Terraform
Education
Roles
Related jobs
-
Senior-level Full TimeIN-AP-Hyderabad3h ago
-
GCP Data Engineer INR 1500K-2400KAlation | Amazon Redshift | Apache Airflow | Apache Flink | Apache KafkaSenior-level Full TimeIN-AP-Hyderabad3h ago
-
GCP Data Engineer INR 1500K-2400KAirflow | Alation | Amazon Redshift | Apache Spark | AzureBest practice development | Code reviews | MentoringSenior-level Full TimeIN-AP-Hyderabad3h ago
-
GCP Data Engineer INR 1500K-2400KAmazon Redshift | Apache Airflow | Apache Flink | Apache Spark | BigQuerySenior-level Full TimeIN-AP-Hyderabad3h ago
-
GCP Data Engineer INR 1500K-2400KAlation | Amazon Redshift | Apache Airflow | Apache Flink | Apache KafkaSenior-level Full TimeIN-AP-Hyderabad3h ago
-
Senior Software Engineer, AI/ML, Google Photos Reminiscing INR 2500K-4460KAudio signal processing | C plus plus | Data Processing | Distributed Systems | Information RetrievalSenior-level Full TimeBengaluru, Karnataka, India5h ago
-
Gen AI Developer Specialist INR 2000K-3500KAPI Gateway | AWS Lambda | Agile | Amazon API | Amazon API GatewayMid-level Full TimeIndia5h ago
-
Analytics Data science and IOT Lead INR 2500K-3500KProbability | PyTorch | Python | SQL | Scikit-learnSenior-level Full TimeIndia5h ago
-
Big Data Lead INR 1200K-2000KAzure Data | Azure Data Lake | Azure Data Lake Storage | CI/CD | Cloud StorageSenior-level Full TimeIndia5h ago
-
Big Data Lead INR 1500K-2000KABAC | AWS | Airflow | Amazon Kinesis | AzureCross-team collaboration | MentoringSenior-level Full TimeIndia5h ago
-
Senior Software Engineer INR 2156K-2800K.NET | AWS | AWS Lambda | Amazon EC2 | Amazon RDSFamily wellbeing benefits | Health and wellbeing benefits | Hybrid work | Paid time offSenior-level Full TimeHyderabad, India6h ago
-
Operations Automation Associate INR 2000K-3500KAPI Integration | Agent Orchestration | CI/CD | Data Governance | Data LineageSenior-level Full TimeMumbai, Maharashtra, India6h ago
-
Test Lead - Data Automation INR 1500K-2209KData Lineage | Data Pipelines | Data Validation | ETL | PythonSenior-level Full TimeIndia6h ago
-
Senior-level Full TimeIndia6h ago
-
Data Engineer (GCH) - Replacement INR 1500K-2000KAWS | AWS Data | AWS Data Build | Amazon Web Services | BashMid-level Full TimeIndia6h ago
-
Senior Software Engineer INR 2000K-3465KAgent Design | Artificial Intelligence | Automation | Azure AI | Azure DevOpsSenior-level Full TimeBengaluru, KA, IN6h ago
-
Senior E2E Service Optimization Engineer (m/f/d) INR 2466K-3500K5G-NR | IMS | LTE | Python | QUICFlat hierarchies | Flexible working hours | Hybrid work | Modern hardware and digital workplace | Open company cultureSenior-level Full TimeAnkara (TUR), Chennai (IND), Communication Management6h ago
-
Senior - AWS Data Engineering INR 3000K-4000KAWS Glue | AWS Lambda | Amazon S3 | Amazon Web Services | Data ModelingSenior-level Full TimeBangalore, Karnataka, India6h ago
-
Software Engineering Professional - GEN AI INR 1500K-2700KAI orchestration | AIOps | APIs | Access Control | Agent FrameworksMid-level Full TimeBengaluru, IN, 5601037h ago
-
Mid-level Full TimeBangalore, Karnataka, India7h ago
-
Manager - Forward Deployed Engineer / AI Engineer INR 1800K-2500KAPIs | Agentic AI | Azure | Azure OpenAI | BERTMid-level Full TimeBangalore, Karnataka, India8h ago
-
Data Engineer INR 1500K-2000KApache Kafka | Apache Spark | BigQuery | Cloud Data | Cloud Data PlatformsMid-level Full TimeIndia9h ago
-
Mid-level Full TimeGurgaon, HR, India9h ago
-
Cloud Support Engineer – Backup & Disaster Recovery INR 800K-1000KAccess Management | Amazon Web Services | Backup | Cloud platform | ComputingCareer pathways | Development programs | Product trainingMid-level Full TimeBangalore, India10h ago
-
Software Engineer III - Data Engineer, Databricks INR 2000K-2800KACID | AWS | Access Controls | Airflow | AlertingSenior-level Full TimeBengaluru, Karnataka, India10h ago