Senior Data Engineer (PySpark, SQL, Databricks)
Tasks
- Build Databricks lakehouse solutions
- Build and manage Delta Lake tables
- Contribute to CI CD deployment automation
- Design scalable data pipelines
- Develop SQL queries and stored procedures
- Implement data quality checks reconciliation and validation
- Implement incremental processing merge upsert and CDC
- Implement medallion architecture bronze/silver/gold
- Migrate legacy workflows to PySpark pipelines
- Optimize Delta tables partitioning and compaction
- Set up governance monitoring lineage and metadata management
- Tune Spark jobs using execution plan optimization
Perks/Benefits
Skills/Tech-stack
ACID | AWS | CI/CD | Change Data Capture | Compaction | Data Capture | Data Governance | Data Lineage | Data Modeling | Data Observability | Data Pipelines | Data Quality | Databricks | Delta Lake | Delta MERGE | Job tuning | Lakehouse Architecture | Medallion Architecture | Partitioning | PySpark | Query Optimization | SQL | Schema Enforcement | Schema evolution | Spark | Spark Job Tuning | Time Travel | Unity Catalog | Upsert
Education
N/A
Roles
Related jobs
-
AWS | Azure | Cloud platform | Data Analysis | Data ManagementAutonomy | Collaborative team environment | Continuous learning | Exposure to modern machine learning technologies | Flexible work scheduleMid-level Full TimeIndia R14h ago
-
Computer Scientist ( Full Stack - Frontend Heavy ) INR 2000K-3500KAWS | Data Structures | Docker | DynamoDB | JavaSenior-level Full TimeBangalore, India R20h ago
-
Computer Scientist 2 – Fullstack (Frontend Heavy) INR 3242K-4580KAccessibility | Babel | CI/CD | CSS | Cloud servicesSenior-level Full TimeNoida, India R20h ago
-
Database Engineering Associate INR 1500K-2000KAWS | Alerting | Cause analysis | Disaster Recovery | ETLFlexible onsite remote schedule | Health and life insurance | Hybrid work model | Paid time off | Pension and retirement benefitsMid-level Full TimeHyderabad, India R1d ago
-
Mid-level Full TimeNoida,Uttar Pradesh,India R1d ago
-
Data Engineer-Mobile Monetization - Big Data, GenAI INR 750K-1200KAI Observability | AWS | Apache Hudi | Apache Iceberg | CI/CDBroadband reimbursement | Catered lunches | Healthcare insurance | Healthy snacks | Hybrid work scheduleMid-level Full TimePune, IN R1d ago
-
Data Engineer INR 2000K-2700KAWS Redshift | Agile | Azure DevOps | Azure Synapse | Azure Synapse AnalyticsHybrid workingMid-level Full TimeIndia R1d ago
-
Sr Forward Deployed Engineer INR 2400K-3900KAPI Integration | AWS | Apache Spark | Artificial Intelligence | AzureSenior-level Full TimeRemote - India R1d ago
-
AWS Glue | AWS IAM | Amazon Redshift | Amazon S3 | Apache AirflowCollaborative environment | Continuous learning | Full-time employment | Fully remote work | Opportunity to work on modern cloud data platformsMid-level Full TimeIndia R1d ago
-
GCP, Digital Analytics, Bigquery/Senior Consultant Specialist INR 2500K-3900KBigQuery | Cloud platform | Data Governance | Data Modeling | Data QualityContinuous professional development | Flexible working | Inclusive and diverse environment | Opportunities for growthSenior-level Full TimeHyderabad, Telangana, India R1d ago
-
Senior AI/ML Engineer INR 2775K-4995KAmazon Web Services | Apache Hadoop | Apache Spark | Azure | Bias MitigationSenior-level Full TimeBengaluru East, Karnataka, India R1d ago
-
Data Engineer INR 3000K-4000KAlteryx | DBT | Data Governance | Data Modeling | Data PipelinesEmployee incentive programs | Flexible work hours | Headspace access | Hybrid work | Mental health daysMid-level Full TimeIndia, Bengaluru, Karnataka R1d ago
-
Senior-level Full TimeBangalore, India R1d ago
-
Data & AI Architect INR 2500K-5000KAI Governance | Agentic AI | Apache Spark | Azure Data | Azure Data FactorySenior-level Full TimeIndia - Remote R1d ago
-
Senior Data Engineer INR 3242K-4580KAI tools | AWS | Airflow | Batch Architecture | DBTRemote work opportunitiesSenior-level Full TimeIndia R2d ago
-
Computer Scientist II (Full Stack) INR 1063K-2500KAPI Versioning | Asynchronous processing | Authentication | Authorization | CI/CDMid-level Full TimeNoida, India R2d ago
-
Senior-level Full TimeNoida, India R2d ago
-
Associate Database Administrator INR 500K-1500KActive Directory | AlwaysOn | Azure | Case management | Cause analysisMid-level Full TimeRemote Pune-Baroda, India R2d ago
-
AWS | Agent systems | Azure | Cloud deployment | Code reviewSenior-level Full TimeBangalore, India R2d ago
-
Sustainability Digital Technology Senior Specialist - Data & AI INR 2500K-4680KAI coding | APIs | Agile | Apache Spark | CI/CDDisability programs | Flexible working hours | Life insurance | Private medical care | Remote workSenior-level Full TimeIN-KA-BANGALORE-NEON BUILDING WEST TOWER, India R2d ago
-
Senior-level Full TimeIndia - Remote R2d ago
-
API Design | Anthropic | Automation | CRM Integrations | CrewAIDistributed work flexibility | Fully remote | High ownership roleMid-level Full TimeIndia R5d ago
-
AI Testing | AI testing frameworks | Adversarial Testing | Bias Assessment | Experiment trackingFlexible contract | Fully remoteSenior-level Contract Full TimeIndia R5d ago
-
AI Engineer - FDE (Forward Deployed Engineer) INR 3242K-4500KAWS | Agent systems | Apache Spark | Azure | DspyRemote workSenior-level Full TimeRemote - India R5d ago
-
Senior BI & AI Engineer INR 3000K-4064KAnomaly Detection | Azure Data | Azure Data Factory | Azure DevOps | DAXESG opportunities | Flexible vacation | Headspace app | Hybrid work model | Mental health daysSenior-level Full TimeIndia, Hyderabad, Telangana R5d ago