Architect (ATC)
Tasks
- Architect Lakehouse solutions with Databricks
- Build incremental data loading frameworks
- Build pipelines with Delta Live Tables and Autoloader
- Create metadata driven ingestion pipelines
- Design batch and real time streaming pipelines
- Develop ETL ELT pipelines with PySpark Scala and SQL
- Develop streaming solutions with Structured Streaming
- Establish data quality frameworks and governance
- Implement Unity Catalog security and access controls
- Implement data modeling using 3NF and dimensional modeling
- Optimize Spark job and compute performance
- Orchestrate workflows and manage deployments with CI CD
- Provide technical leadership and architectural guidance
Perks/Benefits
- N/A
Skills/Tech-stack
Access Control | Airflow | Apache Spark | Autoloader | CI/CD | Data Governance | Data Modeling | Data Quality | Data Warehousing | Databricks | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | ELT | ETL | Fine-Grained Access Control | Incremental loading | Lakehouse | Metadata Driven | Metadata Driven Ingestion | PySpark | SQL | Scala | Structured Streaming | Third Normal Form | Unity Catalog
Education
N/A
Roles
Architect | Data Engineer | Data Platform | Data Platform Architect | Engineer | Platform Architect
Related jobs
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CI/CDLocal connection | Periodic meet ups | Remote work | Structured secure work environment | Support for education and wellnessSenior-level Full TimeJalalabad, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | Cloud Composer | Cloud Dataproc | Cloud MonitoringBudgets for personal growth education and wellness | Legal Compliance Support | Local connection meetups | Remote workSenior-level Full TimeMumbai, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | Bitemporal data modeling | CI/CD | Change Data CaptureDedicated budgets for education | Dedicated budgets for wellness | Innovative projects | Local connection | Periodic meetupsSenior-level Full TimeChennai, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | BigQuery | Change Data Capture | ClickHouseBudget for education | Budget for wellness | Local compliance in India | Periodic team meetups | Personal growth supportSenior-level Full TimeGurgaon Kty., India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAnomaly Detection | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Innovative projects | Local connection | Periodic meet ups | Personal growth budgetSenior-level Full TimeCoimbatore South, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | BigQuery | Bitemporal data modeling | CI/CDEducation budget | Innovative projects | Local connection | Periodic meet ups | Personal growth budgetSenior-level Full TimeJalalabad, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAgentic AI | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Growth budget | Local meetups | Remote work | Wellness budgetSenior-level Full TimeGurgaon Kty., India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | BigQuery | CI/CD | Change Data CaptureEducation budget | Innovative projects | Local compliance | Local connection | Periodic meetupsSenior-level Full TimeAhmedabad, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Innovative projects | Local connection | Periodic meetups | Personal growth budgetSenior-level Full TimeChennai, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | CI/CD | Change Data Capture | Change Data Capture (CDC)Education budget | Local compliance in India | Periodic team meetups | Personal growth budget | Remote workSenior-level Full TimeNew Delhi, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | BigQuery | CI/CD | Change Data CaptureEducation budget | Local compliance support | Periodic team meetups | Personal growth budget | Remote workSenior-level Full TimeKochi, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | CI/CD | Change Data Capture | ClickHouseEducation budget | Innovative projects | Legal presence in India compliance | Local connection meetups | Personal growth budgetSenior-level Full TimeJaipur, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CD pipelinesEducation budget | Innovative projects | Local connection periodic meetups | Personal growth budget | Remote workSenior-level Full TimeTrivandrum, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | BigQuery | CI/CD | Change Data CaptureEducation budget | Local connection meetups | Personal growth budget | Remote work | Structured secure work environmentSenior-level Full TimeIndore, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | Apache pandas | BigQuery | CI/CDInnovative projects | Legal compliance in India | Local connection meetups | Personal growth education and wellness budget | Remote workSenior-level Full TimeKolkata, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CI/CDGrowth education budget | Local connection meetups | Remote work | Secure compliant work environment | Wellness budgetSenior-level Full TimeKolkata, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Innovative projects | Local connection meetups | Personal growth budget | Remote workSenior-level Full TimeAhmedabad, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAgentic AI | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Innovative projects | Legal presence in India | Local connection | Periodic team meetupsSenior-level Full TimeJaipur, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CI/CDEducation budget | Local connection meetups | Professional growth budget | Remote work | Wellness budgetSenior-level Full TimeKochi, India9h ago
-
Data Engineer (GCP) ID56375 INR 2400K-3380KApache Airflow | Apache Spark | CI/CD | Cloud Composer | Cloud MonitoringEducation budget | Local compliance support | Local meetups | Personal growth budget | Remote workSenior-level Full TimeTrivandrum, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | Bitemporal data modelingEducation budget | Legal Compliance Support | Local connection | Meetups | Professional growth budgetSenior-level Full TimeNew Delhi, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | Bitemporal data modelingEducation budget | Innovative projects | Local connection meetups | Personal growth budget | Remote workSenior-level Full TimeMumbai, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CI/CDEducation budget | Innovation projects | Local connection meetups | Personal growth budget | Remote workSenior-level Full TimeHyderabad, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAnomaly Detection | Azure Data | Azure Data Factory | Azure Key Vault | Azure SQLEducation budget | Local connection | Meetups | Personal growth budget | Remote workSenior-level Full TimePune City, India9h ago
-
Data Engineer ID56374 INR 1500K-2300KAzure Data | Azure Data Factory | Azure Key Vault | Azure SQL | CI/CDEducation budget | Innovative projects | Local connection | Periodic meetups | Personal growth budgetSenior-level Full TimeBangalore North, India9h ago