Big Data Lead
Tasks
- Apply AWS best practices for data storage and security
- Automate ETL processes
- Build data pipelines
- Export and import data
- Generate code using Python and Jinja templates
- Implement CI/CD using GitHub
- Implement dbt tests for data validation
- Monitor pipeline runs and reprocess error records
- Optimize SQL queries for performance
- Orchestrate pipelines in sequence or parallel
- Schedule data workflows with Airflow
- Troubleshoot data pipeline errors
- Write Python unit integration and system tests
Perks/Benefits
Skills/Tech-stack
AWS S3 | Amazon ECS | Amazon IAM | Apache Airflow | CI/CD | DBT | Data Pipelines | ETL | GitHub | Jinja | Multithreading | Python | SQL | Snowflake
Education
N/A
Roles
Big Data Engineer | Big Data Lead | Data Engineer | Data Lead | Engineer | Lead
Related jobs
-
Member of Technical Staff - AI/ML Engineer INR 2500K-4500KAnomaly Detection | BM25 | Computer Vision | Document Understanding | Entity ResolutionEquity | Professional developmentSenior-level Full TimeHyderabad, TS, IN16h ago
-
AI-Native Software Developer INR 2500K-4000K.NET | AI Evaluation | AI Orchestration Framework | AI guardrails | AI orchestrationMid-level Full TimeBangalore - Karnataka - India16h ago
-
Senior-level Full TimeMaharashtra, Pune, India17h ago
-
Mid-level Full TimeIN-AP-Hyderabad17h ago
-
Senior-level Full TimeIN-AP-Hyderabad17h ago
-
Senior-level Full TimeIN-AP-Hyderabad17h ago
-
Senior-level Full TimeIN-AP-Hyderabad17h ago
-
Senior-level Full TimeIN-TN-Chennai17h ago
-
Senior-level Full TimeIN-HR-Gurgaon17h ago
-
Mid-level Full TimeBangalore, Karnataka, India18h ago
-
AI Data Engineer III INR 3000K-4000KAgile | Amazon Web Services | Apache Airflow | CI/CD | DBTComprehensive insurance | Education assistance program | Fitness benefits | Flexible working hours | Health checkupsSenior-level Full TimeBengaluru, Karnataka, India18h ago
-
Staff Engineer_SAP Analytics Data Management INR 1500K-2700KABAP | ADSO | Analytics Cloud | CompositeProvider | Data ModelingOnsite 5 days per week | Production rollout support | UAT supportSenior-level Full TimeNoida, India18h ago
-
Staff Data Engineer INR 3000K-4132KAWS | Agile | Amazon Redshift | Amazon S3 | Apache AirflowEducation assistance | Fitness benefits | Flexible working hours | Health checkups | Health insuranceSenior-level Full TimeBengaluru, Karnataka, India18h ago
-
Mid-level Full TimeBangalore, Karnataka, India18h ago
-
Power BI Fabric Engineer INR 2000K-2700KAzure Data | Azure Data Factory | CI/CD | DAX | Data FactoryAnnual excellence awards | Bike to work | Career mentorship | Educational assistance | Employee assistance programmeMid-level Full TimeBengaluru, KA, India19h ago
-
Mid-level Full TimeBangalore, Karnataka, India19h ago
-
Mid-level Full TimeBangalore, Karnataka, India19h ago
-
Big Data Lead INR 1500K-2500KData Architecture | Data Governance | Data Management | Data Management Cloud | Data WarehouseSenior-level Full TimeIndia20h ago
-
Lead Software Engineer -Python or Java, big data INR 3000K-4500KAWS | AWS Lambda | Agile | Amazon EMR | Amazon Elastic Compute ServiceSenior-level Full TimeBengaluru, Karnataka, India20h ago
-
LAM-MTS-AWS+PySparks INR 1200K-2000KAWS Glue | AWS Lambda | AWS Step Functions | Amazon Athena | Amazon CloudWatchMid-level Full TimeNoida, Uttar Pradesh, India20h ago
-
Senior-level Full TimeBengaluru, KA, India20h ago
-
Senior-level Full TimeBengaluru, KA, India20h ago
-
Senior Engineer, Data Engineering INR 1500K-2500KAirflow | Azure | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeBengaluru, KA, India20h ago
-
Solution Integrator (Agentic AI Developer) INR 2475K-4580KAWS | Anthropic Claude | Azure | Chroma | CrewAIEntry-level Full TimeNoida,Uttar Pradesh,India20h ago
-
Mid-level Full TimeBangalore, Karnataka, India R20h ago