Data Engineer
Tasks
- Architect medallion data lakehouse pipelines
- Build data ingestion pipelines for real time and batch
- Conduct research on new data sources and scraping methods
- Create feature store for AI models
- Deploy monitoring alerting and data quality checks
- Develop low latency data APIs and caching layers
- Establish cloud infrastructure and job orchestration
- Implement websocket listeners and streaming infrastructure
- Run nightly batch pipelines for retraining datasets
Perks/Benefits
- N/A
Skills/Tech-stack
API Integration | AWS | Aiohttp | Airflow | Asyncio | BeautifulSoup | Dagster | Data Modelling | Docker | GCP | Linux | PostgreSQL | Prefect | Python | REST API | Redis | SQL | Scrapy | Streaming Data | Web Scraping | WebSockets | Webhooks
Education
N/A
Roles
Related jobs
-
Data Scientist/Data Engineer/Data Analyst INR 1500K-2000KA/B | A/B Testing | AWS Glue | Amazon Redshift | Amazon S3Mid-level Full TimeChennai City Corporation, India2h ago
-
Mid-level Full TimeHyderabad, India2h ago
-
Mid-level Full TimeHyderabad, India2h ago
-
Mid-level Full TimeBangalore North, India2h ago
-
Data Engineer INR 400K-1200KApache Airflow | Apache Spark | Azure Data | Azure Data Factory | CI/CDSenior-level Full TimeCoimbatore, India2h ago
-
AI Engineer : Mumbai INR 2500K-4170KAPI Development | AWS | Agentic AI | Ansible | AzureEquity compensation | Mentoring | Training programsSenior-level Full TimeMaharashtra, Mumbai, India2h ago
-
Mid-level Full TimeIN-KA-Bangalore3h ago
-
Data Engineer INR 2000K-2400KAirflow | Apache Spark | Cloud platform | Data Modeling | Google BigQuerySenior-level Full TimeIN-KA-Bangalore3h ago
-
Senior Engineer - PowerBI with Databricks INR 2400K-3500KDashboard Design | Data Modeling | Data Quality | Data Visualization | Data pipelineCareer growth | Mentorship | Paid certification support | Recognition and rewards | Snack allowanceSenior-level Full TimeHyderabad, TS, India4h ago
-
Senior Staff Engineer, Data Science INR 2000K-3380KAWS | Azure | Classification | Cloud platform | Data AnalysisSenior-level Full TimeRemote, India R4h ago
-
Senior Staff Engineer, Data Science INR 2000K-3380KAWS | Azure | Classification | Continuous Improvement | Data AnalysisContinuous improvement culture | Global team environment | MentorshipSenior-level Full TimeRemote, India R4h ago
-
Senior BI Engineer - Power BI INR 2400K-3380KAzure SQL | DAX | Data Modeling | Data Warehousing | DatabricksCareer growth | Mentorship | Recognition rewards | Snack zone | Training supportSenior-level Full TimeHyderabad, TS, India4h ago
-
Lead Engineer - Power BI INR 2040K-3500KAzure SQL | DAX | Data Warehousing | Databricks | ETLCareer growth | Certification cost support | Mentorship | Recognition and rewards | SnacksSenior-level Full TimeHyderabad, TS, India4h ago
-
Python / Pyspark / GCP INR 2156K-2700KBigQuery | Cloud Bigtable | Cloud platform | Cucumber | Google CloudEntry-level Full TimeBangalore, Karnataka, India6h ago
-
Software Engineer III - Data Engineer INR 2000K-2755KAWS | Apache Airflow | Azure | Big Data | Data GovernanceSenior-level Full TimeMumbai, Maharashtra, India6h ago
-
Mid-level Full TimeHyderabad, Telangana, India6h ago
-
Lead Software Engineer - Pyspark / Bigdata INR 1800K-3200KAgile | Application Resiliency | Application development | Big Data | CI/CDSenior-level Full TimeBengaluru, Karnataka, India7h ago
-
Associate III- Business Intelligence-HO & SUPPORT-CVM CoE - Corporate Centre of Excellence INR 1000K-1500KAdvanced Excel | Caret | Classification | Clustering | Data AnalysisMid-level Full TimeBangalore, Karnataka, India7h ago
-
Advanced Excel | Caret | Classification | Clustering | Data AnalysisMid-level Full TimeBangalore, Karnataka, India7h ago
-
Senior-level Full TimeIndia8h ago
-
Mid-level Full TimeBangalore, Karnataka, India8h ago
-
Mid-level Full TimeBangalore, Karnataka, India8h ago
-
DE&A - Core - Advanced Data Engineering - Data Modeling INR 1500K-2700KAPIs | BI Publisher | Business Suite | Cause analysis | Data LakesMid-level Full TimeIndia8h ago
-
Sr Lead Software Engineer INR 2500K-4000KAWS | AWS Security | Agentic AI | Amazon Bedrock | Context windowSenior-level Full TimeMumbai, Maharashtra, India8h ago
-
Data Science Engineer INR 2400K-3500KCloud Database | Data Modeling | Database Design | ELT | ETLCommercial awareness | Onshore Offshore model experience | UK client experienceSenior-level Full TimeMumbai, MH, India9h ago