Principal Data Engineer
Lahore, Pakistan
Tkxel
Looking for Top Software Development Company in USA? We Engineer Software Solutions for Enterprises, Growth Companies & Startups.Job Description:
We are seeking an experienced Data Engineer to the
design, development, and optimization of our client data infrastructure. This
role requires deep expertise in cloud technologies (primarily Azure, AWS is a plus) and data engineering best practices, with additional
experience in Apache Spark and Databricks for large-scale data
processing. The Data Engineer will work closely with data scientists, analysts,
and other stakeholders to create scalable and efficient data systems that
support advanced analytics and business intelligence. Additionally, this role involves
mentoring junior engineers and driving technical innovation within the data
engineering team.
Key Responsibilities:
- Collaborate
with Solution Architects: Work with Big Data Solution Architects to
design, prototype, implement, and optimize data ingestion pipelines,
ensuring effective data sharing across business systems.
- ETL/ELT
Pipeline Development: Build and optimize ETL/ELT pipelines and
analytics solutions using a combination of cloud-based technologies, with
an emphasis on Apache Spark and Databricks for large-scale
data processing.
- Data
Processing with Spark: Leverage Apache Spark for distributed
data processing, data transformation, and analytics at scale. Experience
with Databricks for optimized Spark execution is highly desirable.
- Production-Ready
Solutions: Ensure data architecture, code, and processes meet
operational, security, and compliance standards, making solutions
production-ready in cloud environments.
- Project
Support & Delivery: Actively participate in project and production
delivery meetings, providing technical expertise to resolve issues quickly
and ensure successful project execution.
- Database
Management: Manage both SQL (e.g., PostgreSQL, MySQL) and NoSQL (e.g., DynamoDB, MongoDB) databases, ensuring data is efficiently stored,
retrieved, and queried.
- Real-Time
Data Processing: Implement and maintain real-time data streaming
solutions using tools such as Apache Kafka, AWS Kinesis, or
other technologies for low-latency data processing.
- Cloud
Monitoring & Automation: Use monitoring and automation tools
(e.g., AWS CloudWatch, Azure Monitor) to ensure efficient
use of cloud resources and optimize data pipelines.
- Data
Governance & Security: Implement best practices for data
governance, security, and compliance, including data
encryption, access controls, and audit trails to meet regulatory
standards.
- Collaboration
with Stakeholders: Work closely with data scientists, analysts, and
business teams to align data infrastructure with strategic business
objectives and goals.
- Documentation:
Maintain clear and detailed documentation of data models, pipeline
processes, and system architectures to support collaboration and
troubleshooting.
Requirements
- 5+
years of experience as a Data Engineer, with strong expertise in
cloud-based data warehousing, ETL pipelines, and large-scale data
processing.
- Proficiency
with cloud technologies, with experience in platforms like Azure or AWS.
- Hands-on
experience with Apache Spark for distributed data processing and
transformation. Experience with Databricks is highly desirable.
- Strong SQL skills and experience with relational databases (e.g., PostgreSQL, MySQL) as well as NoSQL databases (e.g., MongoDB, DynamoDB).
- Proficient
in Python for data processing, automation tasks, and building data
workflows.
- Experience
with PySpark for large-scale data engineering, particularly in Spark
clusters or Databricks.
- Experience
in designing and optimizing data warehouse architectures, ensuring
optimal query performance in large-scale environments.
- A
strong understanding of data governance, security, and compliance best practices, including encryption, access control, and data privacy.
Preferred Qualifications:
- Bachelor’s
degree in Computer Science, Engineering, or a related field.
- Certifications in Data Engineering from cloud providers (e.g., AWS Certified
Big Data - Specialty, Microsoft Certified: Azure Data Engineer
Associate) are a plus.
- Experience
with advanced data engineering tools and platforms such as Databricks, Apache Spark, or similar distributed computing technologies
* Salary range is an estimate based on our AI, ML, Data Science Salary Index 💰
Tags: Architecture AWS Azure Big Data Business Intelligence Computer Science Databricks Data governance Data pipelines Data warehouse Data Warehousing DynamoDB ELT Engineering ETL Kafka Kinesis MongoDB MySQL NoSQL Pipelines PostgreSQL Privacy PySpark Python RDBMS Security Spark SQL Streaming
More jobs like this
Explore more career opportunities
Find even more open roles below ordered by popularity of job title or skills/products/technologies used.