G09 - Data Engineer

We are seeking a highly skilled and experienced Data Engineer to design, build, and optimize end-to-end data architecture and pipelines for social service platforms such as Client360 and Household360. This role involves leading efforts in data modeling, pipeline development, API integration, data security, and cross-agency data synchronization, ensuring secure and efficient access to critical citizen information.

Key Responsibilities:

1. Data Architecture & Pipeline Development

• Design and implement scalable and reusable data models to support social service integration (e.g., Client360, Household360).
• Build and manage ETL/ELT pipelines for legacy system migration and real-time data synchronization across agencies.
• Develop and maintain batch and streaming data pipelines for cross-agency data sharing.
• Implement data validation frameworks and monitoring systems to ensure high data quality.

2. Integration Development & API Management

• Design and develop RESTful APIs and system integrations for solutions like OneCV, CaseConnect, ComLink+.
• Develop and maintain detailed API documentation, including interface contracts and change logs.
• Ensure API performance, security, availability, and scalability through structured testing and monitoring.

3. Data Security & Compliance

• Enforce data protection measures, including field-level encryption, role-based access controls, and secure data transfer.
• Ensure compliance with IM8, Security Standards, and other relevant regulatory frameworks.
• Maintain audit trails, implement data masking, and classify sensitive data appropriately.
• Monitor and enforce data policies across environments.

4. Performance Optimization & Monitoring

• Tune query performance and batch processing to support near real-time analytics.
• Monitor and optimize real-time data synchronization processes.
• Develop and maintain performance dashboards to track pipeline health and system metrics.
• Ensure high system reliability, uptime, and service level objectives (SLOs) are met.

5. Cross-Team Collaboration

• Collaborate closely with business analysts, solution architects, DevOps, and frontend/backend developers.
• Participate in Agile ceremonies, including sprint planning, backlog grooming, and retrospectives.
• Conduct code reviews, engage in pair programming, and foster technical excellence within the team.
• Provide expertise and support on data integration and architecture.

6. Quality Assurance & Documentation

• Maintain clear and up-to-date technical documentation, including system design and data flow diagrams.
• Develop automated test cases to validate data transformations and integration logic.
• Conduct end-to-end data accuracy validation across source and destination systems.
• Create troubleshooting playbooks and resolution procedures to support operational teams.

Requirements:

• Bachelor's or Masters degree in Computer Science, Information Systems, or related discipline.
• 7+ years of experience in data architecture, data engineering, or system integration roles.
• Proficient in ETL tools, SQL, data modeling, and real-time stream processing frameworks (e.g., Kafka, Spark, Flink).
• Strong experience with API development and integration standards (REST, JSON, OAuth2, OpenAPI).
• Familiarity with systems integration, services, and compliance requirements.
• Solid understanding of data security, access control, and compliance frameworks (e.g., IM8, ISO 27001).
• Experience in cloud platforms (e.g., Azure, AWS, or GCC), CI/CD pipelines, and monitoring tools.
• Excellent communication and documentation skills.