DATA Architect - Lead Systems Engineer
Tasks
- Automate notebook and job deployments
- Build data pipelines for analytics reporting and machine learning
- Build end-to-end data pipelines
- Develop and optimize SQL queries and dashboards
- Implement data governance with Unity Catalog
- Implement data masking encryption and audit logging
- Implement incremental processing with Delta Lake
- Implement lakehouse with Delta Lake and Apache Spark
- Integrate data workflows into CI CD using DevOps principles
- Lead data engineering team
- Manage fine grained access controls
- Manage source control with GitLab
- Optimize cluster utilization and job scheduling
- Own GCP data lake architecture
- Profile and refactor data processing code
- Set up retrieval and indexing for RAG systems
- Track data lineage
- Tune SQL workloads and caching
Perks/Benefits
- N/A
Skills/Tech-stack
Agile | Apache Airflow | Apache Kafka | Apache Spark | Audit Logging | CI/CD | Cloud Storage | Cloud platform | Data Engineering | Data Governance | Data Lineage | Data masking | Databricks Unity | Databricks Unity Catalog | Delta Lake | ETL | Embeddings | Encryption | Git | GitLab | Google Cloud | Google Cloud Platform | Google Cloud Storage | Google Pub/Sub | Incremental processing | Jenkins | Looker Studio | Machine Learning | PostgreSQL | Power BI | Pub/Sub | PySpark | Python | RAG | SQL | Scala | Terraform | Unity Catalog | Vector Databases
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Senior-level Full TimeHyderabad, IN, 50008114h ago
-
Senior Software Engineer (Infra) INR 2400K-3500KC++ | Distributed Systems | Java | Networking | PythonSenior-level Full TimeBengaluru, India15h ago
-
Senior Software Engineer - Search INR 2500K-4200KInformation Retrieval | Language Models | Language Processing | Large Language Models | Machine LearningSenior-level Full TimeBengaluru, India15h ago
-
Senior Software Engineer - Data Platform INR 2500K-3584KAWS | Agentic AI | Apache Airflow | Apache Druid | Apache FlinkAccident insurance | Commuter benefits | Dental insurance | Disability insurance | Financial wellness supportSenior-level Full TimeBengaluru, India R15h ago
-
Mid-level Full TimeChennai, Tamil Nadu, India16h ago
-
Manager AI & Analytics INR 1800K-3000KAPIs | Amazon QuickSight | Artificial Intelligence | Business Intelligence | Data GovernanceMid-level Full TimeIndia Office20h ago
-
Senior-level Full TimeChennai, Tamil Nadu, India1d ago
-
Data Engineering Manager - GCP INR 1500K-2500KAirflow | Apache Spark | BigQuery | CI/CD | Cloud ComposerSenior-level Full TimeHyderabad, TS, India1d ago
-
Anthropic Claude | Application Insights | Asynchronous programming | Automated testing | Azure Application InsightsFlexible working environment | Long-term engagement | Mentorship opportunities | Remote workSenior-level Contract Full TimeIndia R1d ago
-
Mid-level Full TimeBangalore, India Office (BANGALORE)1d ago
-
Senior-level Full TimeHyderabad, Noida, Chennai, India1d ago
-
Manager, Marketing Analytics and Data Science INR 900K-1000KA/B | A/B Testing | B testing | Data Governance | Data ModelingMid-level Full TimeBangalore, India1d ago
-
Lead Salesforce Full Stack AI Engineer INR 3715K-5449KAPI Design | Agile | Apex | Automated testing | Azure DevOpsFlexible schedule | Generous vacation days | Health coverage | Parental leave | Volunteer timeSenior-level Full TimeBangalore, India1d ago
-
AI Foundry | AWS | AWS SageMaker | Agentic AI | Amazon Bedrock401k match | Career development resources | Dental insurance | Flexible work arrangements | Health Savings Account medical planSenior-level Full TimeIndia1d ago
-
Engineer, Storage and Data Protection INR 1500K-2600KAlerting | Ansible | Automation | Backup | BashMid-level Full TimeGurugram, Haryana1d ago
-
AWS Glue | AWS Lambda | AWS Step Functions | Amazon EC2 | Amazon EMRSenior-level Full TimeNoida, India1d ago
-
Associate Engineer - Data engineer INR 1000K-1700KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure DatabricksCollaborative work environment | Professional development | Training opportunitiesMid-level Full TimeBangalore - Karnataka, India - Kalyani …1d ago
-
Senior-level Full TimeGurugram, India1d ago
-
Mid-level Full TimeHyderabad, India1d ago
-
AI Transformation Engineer INR 2000K-4408KAPI Integration | Agent Frameworks | Agents SDK | Automated testing | Backend DevelopmentMid-level Full TimeBengaluru, India1d ago
-
Senior Forward Deployed Engineer INR 1500K-2800KAI APIs | AWS | Apache Spark | Azure | CI/CDCustomer-facing role | Travel 20 percent of the timeSenior-level Full TimeBengaluru, India1d ago
-
Senior Software Engineer, AI/ML INR 2000K-5000K3D mathematics | Agent systems | Autonomous Agents | C plus plus | Computational GeometrySenior-level Full TimeBengaluru, Karnataka, India1d ago
-
Customer Engineering Manager, Data Analytics, Google Cloud INR 2500K-4500KAI machine learning | Cloud Architecture | Cloud platform | Data Governance | Data LakeSenior-level Full TimeBengaluru, Karnataka, India; Mumbai, Maharashtra, India1d ago
-
Senior Software Engineer - BLR/HYD INR 2000K-4500KArtificial Intelligence | Cloud services | Debugging | Distributed Systems | Machine LearningSenior-level Full TimeIndia, Karnataka, Bangalore; India, Telangana, Hyderabad1d ago
-
Senior Data Engineer INR 3000K-4000KAmazon Web Services | Apache Spark | Cloud platform | Continuous Delivery | Data GovernanceSenior-level Full TimeBangalore, India; Chennai, India; Hyderabad, India; …1d ago