Big Data Engineer
Tasks
- Design batch and real-time data pipelines
- Develop distributed data processing solutions
- Document data processes and architectures
- Ensure data governance security and data quality
- Implement real-time streaming pipelines
- Ingest, transform, and integrate data
- Manage Hadoop distributed storage
- Monitor and troubleshoot data workflows
- Optimize pipeline performance and scalability
- Orchestrate ETL workflows
- Query data using Hive and Impala
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Apache Hadoop | Apache Hive | Apache Impala | Apache Kafka | Apache NiFi | Apache Spark | Azure | CI/CD | Cloudera Data Platform | Compliance | Data Governance | Data Modeling | Data Quality | Data Security | Data Warehousing | Data platform | Docker | ETL | GCP | HDFS | Java | Kubernetes | PySpark | Python | SQL | Scala
Education
Roles
Related jobs
-
Artificial Intelligence/Machine Learning Engineer USD 150K-250KAWS | Azure | Cloud platform | Computer Vision | Continuous integrationCompany-paid trips | Easy loan facility | Gym membership | Health insurance | Leaves as per company policySenior-level Full TimeLahore, Punjab, Pakistan17h ago
-
Big Data Engineer INR 2000K-2000KApache Hadoop | Apache Hive | Apache Impala | Apache Kafka | Apache NiFiMid-level Full TimeKarachi, Sindh17h ago
-
Senior Data Scientist II EUR 54K-84KAB Testing | BigQuery | Data Mining | Deep learning | HadoopFitness reimbursements | Flexible work schedule | Healthcare benefits | Remote work days | Unlimited vacation daysSenior-level Full TimePakistan1d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous work culture | Entrepreneurial-minded team | Friendly and laid-back atmosphere | Hand off managementMid-level Full TimeKarachi, Pakistan1d ago
-
Data Engineer USD 94K-145KCloud Data Integration | Data Modeling | Data integration | ETL | Oracle CloudMid-level ContractKarachi, Sindh1d ago
-
Data Engineer USD 94K-145KCloud Data Integration | Data Modeling | Data integration | ETL | Oracle CloudMid-level ContractLahore, Punjab1d ago
-
Data Engineer USD 151K-208KAWS Bedrock | Airflow | Amazon Kinesis | Amazon Redshift | BigQueryEducational courses | Medical | Time offSenior-level Full TimeLahore, Punjab, Pakistan1d ago
-
Data Engineer, Tech USD 100K-144KData Modeling | Data Quality | Data Warehousing | Database Indexing | ETLMid-level Full TimeLahore, Pakistan2d ago
-
Data Engineer / Senior Data Engineer USD 120K-180KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeLahore, Punjab4d ago
-
Data Engineer / Senior Data Engineer USD 120K-180KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake StorageSenior-level Full TimeKarachi, Sindh4d ago
-
Senior Data Engineer (AWS) USD 139K-209KAWS Glue | AWS Lambda | AWS VPC | Amazon RDS | Amazon RedshiftCollaborative team | Continuous improvement | Remote workSenior-level Full TimeLahore, Punjab5d ago
-
Senior Data Engineer (AWS) USD 139K-209KAWS Glue | AWS Lambda | AWS VPC | Amazon RDS | Amazon RedshiftRemote workSenior-level Full TimeKarachi, Sindh5d ago
-
Data Engineer USD 98K-133KCloud Data Integration | Data Modeling | Data Quality | Data Warehousing | Data integrationMid-level ContractKarachi, Sindh5d ago
-
Data Engineer USD 98K-133KCloud Data Integration | Data Modeling | Data Warehousing | Data integration | DenormalizationOffshore delivery model collaborationMid-level ContractLahore, Punjab5d ago
-
AWS CloudFormation | AWS Glue | Amazon Kinesis | Amazon Redshift | Apache AirflowRemote workMid-level Full TimePakistan - Remote R6d ago
-
Senior Data Engineer (AWS / PLSQL / Oracle) INR 3750K-6250KAWS Lambda | Amazon Redshift | Amazon S3 | Amazon Web Services | Data ModelingHybrid workSenior-level Full TimeKarachi, Sindh6d ago
-
Senior Data Engineer (AWS / PLSQL / Oracle) INR 3750K-6250KAWS | Amazon Redshift | Data Processing | Data Transformation | Data WarehousingHybrid workSenior-level Full TimeLahore, Punjab6d ago
-
Lead Data Engineer (AWS / PLSQL / PL/PGSQL) INR 1500K-4000KAWS Glue | AWS Lambda | AWS Redshift | Amazon S3 | Amazon Web ServicesHybrid work | Technical leadershipSenior-level Full TimeIslamabad, Federal Capital6d ago
-
Lead Data Engineer (AWS / PLSQL / PL/PGSQL) INR 1500K-4000KAWS Glue | AWS Lambda | Amazon Redshift | Amazon S3 | CI/CDSenior-level Full TimeKarachi, Sindh6d ago
-
Senior Data Engineer (AWS / PLSQL / Oracle) INR 3750K-6250KAWS | AWS Lambda | AWS S3 | Amazon Redshift | Data ModelingHybrid workSenior-level Full TimeIslamabad, Federal Capital6d ago
-
Lead Data Engineer (AWS / PLSQL / PL/PGSQL) INR 1500K-4000KAWS Glue | AWS Lambda | Amazon Redshift | Amazon S3 | Amazon Web ServicesHybrid workSenior-level Full TimeLahore, Punjab6d ago
-
Mid-level Full TimeLahore, Punjab, Pakistan6d ago
-
AWS AI/ML Practice Lead CAD 146K-202KAPI Gateway | AWS API | AWS API Gateway | AWS CDK | AWS EventBridgeSenior-level Full TimeLahore, Punjab, Pakistan7d ago
-
Senior Data Engineer GBP 50K-70KApache Kafka | Azure | Azure Data | Azure Data Factory | Azure EventSenior-level Full TimeLahore, Pakistan8d ago
-
Application Programming | Application Programming Interfaces | Azure Data | Azure Data Factory | Azure SQLSenior-level Full TimeKarachi, Pakistan8d ago