Senior Data Engineer – Data Quality & Observability
Tasks
- Build automated validation checks
- Create validation processes for completeness accuracy integrity consistency reconciliation freshness anomaly detection
- Define and maintain data lineage
- Define remediation workflows and quality metrics
- Design and implement scalable data quality framework
- Develop data quality rules as code
- Develop reporting for data quality trends and health metrics
- Implement data observability monitoring and alerting
- Implement data validation solutions
- Improve data engineering practices for reliable solutions
- Integrate validation checks into CI CD pipelines
- Investigate data issues and perform root cause analysis
Perks/Benefits
- Continuous improvement opportunities
- Cross-functional collaboration
- Fully remote work
- Ownership over data reliability strategy
Skills/Tech-stack
Alerting | Amazon Aurora | Amazon Aurora PostgreSQL | Amazon Redshift | Anomaly Detection | Aurora PostgreSQL | Batch Processing | CI/CD | Dashboards | Data Governance | Data Lineage | Data Modeling | Data Observability | Data Reconciliation | Data Validation | ETL | Git | Great Expectations | Monitoring | Python | SQL | Soda
Education
N/A
Roles
Related jobs
-
Blue-Green Deployment | Blue/green | CI/CD | CUDA | CUDA optimizationFully remote | Remote work flexibilitySenior-level Full TimeSouth Africa R5d ago
-
Data Engineer ZAR 396K-456KAPI Development | AWS | Apache Spark | Automation | CI/CDCareer development opportunities | Employee discounts | Hybrid working | Paid sabbaticals | Paid time offSenior-level Full TimeCape Town, Western Cape, South Africa R7d ago
-
Principal Clinical Data Engineer - based in South Africa ZAR 663K-663K21 CFR | 21 CFR Part 11 | CFR Part 11 | Cause analysis | Clinical dataSenior-level Full TimeSouth Africa-Bloemfontein-Remote R10d ago
-
Airflow | Amazon Redshift | BigQuery | CI/CD | ChromaDBFlexible working hours | Remote workSenior-level Full TimeSouth Africa - Remote R11d ago
-
Senior Clinical Data Engineer ZAR 663K-663K21 CFR | 21 CFR Part 11 | CFR Part 11 | Cause analysis | Clinical ResearchSenior-level Full TimeSouth Africa-Bloemfontein-Remote R12d ago
-
Senior Data Platform Consultant ZAR 600K-730KAWS | Azure | CI/CD | Classification | Data EngineeringHybrid workSenior-level Full TimeCape Town, South Africa R28d ago
-
AWS | Amazon Redshift | Apache Airflow | Azure | BigQueryCareer growth opportunities | Collaborative culture | Continuous learning | Flexible working arrangements | Fully remoteSenior-level Full TimeSouth Africa - Remote R28d ago
-
Database Engineer / Database Administrator ZAR 464K-504KAWS Aurora | AWS RDS | Agile | Automation | Database AdministrationAdvancement opportunities | Annual tutoring stipend for children | Employee development program stipend | Fertility/adoption assistance | Mentorship programMid-level Full TimeRemote, South Africa R1mo ago
-
Senior Full Stack Engineer (NLP) ZAR 600K-800KAPI Integration | CSS | Database Schema | Database Schema Design | DjangoAnnual leave | Education stipend | Employee assistance program | Internet benefits | Laptop providedSenior-level Full TimeRemote, South Africa R1mo ago