Big Data Lead
Tasks
- Build ETL/ELT pipelines
- Build streaming data pipelines
- Collaborate on master data management and metadata
- Design data ingestion pipelines
- Develop data models
- Document data sources transformations and runbooks
- Implement CDC micro batch processing
- Implement automated testing
- Implement compliant access with RBAC and ABAC
- Implement data quality checks
- Implement security and privacy controls like masking and tokenization
- Lead complex integrations and migrations
- Lead project execution and cross team alignment
- Optimize storage and compute cost
- Set up observability lineage metrics and alerts
- Write SQL and Python transformations
Perks/Benefits
Skills/Tech-stack
ABAC | AWS | Airflow | Amazon Kinesis | Azure | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Azure RBAC | CDC | CI/CD | DBT | Data Factory | Data Lineage | Data Quality | Data Testing | Data Vault | Delta Lake | Dimensional Modeling | ELT | ETL | Event Hubs | Kafka | Observability | Parquet | Prefect | Python | RBAC | SQL | Terraform
Education
Related jobs
-
Software Engineer INR 1300K-2000KAgile | Cloud services | Data Modeling | Database Design | Design PatternsMid-level Full TimePune, India4h ago
-
Senior-level Full TimeIN-KA-Bangalore8h ago
-
Senior-level Full TimeIN-AP-Hyderabad8h ago
-
GCP Data Engineer INR 1500K-2400KAlation | Amazon Redshift | Apache Airflow | Apache Flink | Apache KafkaSenior-level Full TimeIN-AP-Hyderabad8h ago
-
GCP Data Engineer INR 1500K-2400KAirflow | Alation | Amazon Redshift | Apache Spark | AzureBest practice development | Code reviews | MentoringSenior-level Full TimeIN-AP-Hyderabad8h ago
-
GCP Data Engineer INR 1500K-2400KAmazon Redshift | Apache Airflow | Apache Flink | Apache Spark | BigQuerySenior-level Full TimeIN-AP-Hyderabad8h ago
-
GCP Data Engineer INR 1500K-2400KAlation | Amazon Redshift | Apache Airflow | Apache Flink | Apache KafkaSenior-level Full TimeIN-AP-Hyderabad8h ago
-
Senior Software Engineer, AI/ML, Google Photos Reminiscing INR 2500K-4460KAudio signal processing | C plus plus | Data Processing | Distributed Systems | Information RetrievalSenior-level Full TimeBengaluru, Karnataka, India10h ago
-
Senior-level Full TimeIndia10h ago
-
Workplace Engineering Track Specialist INR 1500K-3000KAWS | Azure | Firewall | Google Cloud | LAN/WANSenior-level Full TimeIndia10h ago
-
Gen AI Developer Specialist INR 2000K-3500KAPI Gateway | AWS Lambda | Agile | Amazon API | Amazon API GatewayMid-level Full TimeIndia11h ago
-
Data Integration Lead INR 1500K-2800KAgile | Oracle Database | Oracle Forms | Oracle Reports | Oracle SQLSenior-level Full TimeIndia11h ago
-
Analytics Data science and IOT Lead INR 2500K-3500KProbability | PyTorch | Python | SQL | Scikit-learnSenior-level Full TimeIndia11h ago
-
Big Data Lead INR 1200K-2000KAzure Data | Azure Data Lake | Azure Data Lake Storage | CI/CD | Cloud StorageSenior-level Full TimeIndia11h ago
-
Big Data Lead INR 1500K-2000KABAC | Amazon Kinesis | Apache Airflow | Azure Data | Azure Data FactorySenior-level Full TimeIndia11h ago
-
Information Delivery & Visualization Lead INR 800K-1500KData Aggregation | Data Blending | Data Cleansing | Data Modeling | Database queryingSenior-level Full TimeIndia11h ago
-
Senior Software Engineer INR 2156K-2800K.NET | AWS | AWS Lambda | Amazon EC2 | Amazon RDSFamily wellbeing benefits | Health and wellbeing benefits | Hybrid work | Paid time offSenior-level Full TimeHyderabad, India11h ago
-
DC Engineer Specialist INR 1500K-3000KAWS | Cloud platform | Google Cloud | Google Cloud Platform | LAN/WANMid-level Full TimeIndia11h ago
-
Operations Automation Associate INR 2000K-3500KAPI Integration | Agent Orchestration | CI/CD | Data Governance | Data LineageSenior-level Full TimeMumbai, Maharashtra, India11h ago
-
Test Lead - Data Automation INR 1500K-2209KData Lineage | Data Pipelines | Data Validation | ETL | PythonSenior-level Full TimeIndia11h ago
-
Data Integration Lead INR 1500K-2000KCI/CD | Control-M | DataStage | Deployment | ETLClient facing communication | Written and verbal communicationSenior-level Full TimeIndia11h ago
-
JAVA Data Engineering/Associate Director Software Engineering INR 1400K-2200KApache Beam | Apache Flink | Apache Kafka | Apache Spark | Data PipelinesContinuous professional development | Flexible workingMid-level Full TimePune, Maharashtra, India R11h ago
-
Senior-level Full TimeIndia11h ago
-
Data Integration Lead INR 800K-2000KAutosys | Bitbucket | CI/CD | Data Warehousing | Dimensional ModelingSenior-level Full TimeIndia11h ago
-
Data Engineer (GCH) - Replacement INR 1500K-2000KAWS | AWS Data | AWS Data Build | Amazon Web Services | BashMid-level Full TimeIndia11h ago