Senior Data Engineer – Databricks Expert

Hyderabad, India

TechVedika

We are an Artificial Intelligence (AI) focused product engineering company providing our customers with futuristic solutions powered by 'humanised' AI harnessing Big Data Analytics, IoT and Web/Mobility services.

View all jobs at TechVedika

Apply now Apply later

Company Description

We are an Artificial Intelligence (AI) focused product engineering company, providing our customers in healthcare, retail & e-commerce, manufacturing, and hospitality sectors with cutting-edge products & solutions, harnessing Big Data Analytics, Vision Analytics, and IoT.

Ever since our inception in March 2010, Tech Vedika has been

  • Great Place To Work Certified™(May 2022-May 2023) Organization 
  • Top 50 I Mid-Size India’s Best Workplaces for Women 2022 !
  • Top 10 Most Disruptive Face & Image Recognition Solution Providers’2020 – Analytics Insights
  • Top 10 Healthcare Analytics Solution Providers’ 2019- Healthcare Outlook Magazine
  • Top 20 most amazing AWS Service Providers – CIO Review India 2018

We strive for simple, elegant tech solutions to perform complex tasks. As a scalable technology partner, we enable organizations to improve operational efficiency and unleash new business potential

 

Job Description

We are seeking a Senior Data Engineer with in-depth knowledge of Databricks and Unity Catalog to serve as the subject matter expert for all things Databricks within our organization. This role requires deep expertise in Databricks, including CI/CD setup for data bricks, data lineage through Unity Catalog, and strong proficiency in ETLSQL, and modern data engineering practices. You will be the go-to person for designing, implementing, and optimizing data solutions with Databricks.

 

Key Responsibilities:

·         Serve as the point of contact and subject matter expert for all Databricks-related activities, including architecture, development, and operational best practices.

·         Design, develop, and manage ETL/ELT pipelines in Databricks using Python (PySpark), integrating various data sources to support business operations.

·         Leverage Unity Catalog to ensure data lineagesecurity, and governance are properly managed across the Databricks environment.

·         Implement and maintain CI/CD pipelines for Databricks, ensuring smooth deployments, version control, and automation using Git and other DevOps tools.

·         Build scalable data architectures, including Data LakesLakehouses, and Data Warehouses, ensuring efficient data management and accessibility.

·         Configure and optimize Databricks clustersjobs, and workflows for both batch and streaming data processing to handle large-scale datasets.

·         Stay up-to-date with the latest Databricks features and advancements, continuously enhancing our data engineering practices.

·         Collaborate with cross-functional teams to implement data governance and ensure compliance with security and industry regulations.

·         Monitor and tune Databricks workloads to ensure high performance and scalability, adapting to business needs as required.

·         Provide guidance and mentorship to junior engineers, ensuring adherence to best practices and fostering a collaborative environment.

Qualifications

Qualifications:

·         5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.

·         Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.

·         Experience building and optimizing ETL pipelines using tools like Azure Data FactoryInformatica, or similar.

·         Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.

·         Expertise in SQL development and performance tuning for large-scale datasets.

·         Knowledge of the Azure ecosystem, including data services like Azure Data Lake and Azure Storage.

·         Ability to work with both batch and streaming data processing pipelines.

·         Experience with data modeling and dimensional design (e.g., star schema).

·         Good understanding of data governance, compliance, and security best practices.

·         Excellent communication and problem-solving skills, with the ability to manage multiple priorities.

·         Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.

Additional Information

At Tech Vedika, we are looking for talented individuals who want to work with driven people. Attain success while working on interesting projects with a culturally diverse group of individuals.

Perks & Benefits: 

  • Health Insurance
  • Meal Vouchers
  • Learning Aids
  • Client/Customer Interactions
  • Working with great minds 

If you want an exciting and dynamic career with unlimited growth potential, then Tech Vedika is the place for you!

Apply now Apply later

* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰

Job stats:  0  0  0
Category: Engineering Jobs

Tags: Architecture AWS Azure Big Data CI/CD Data Analytics Databricks Data governance Data management DevOps E-commerce ELT Engineering ETL Git Informatica Pipelines PySpark Python Security Spark SQL Streaming

Perks/benefits: Career development Health care Startup environment Team events

Region: Asia/Pacific
Country: India

More jobs like this