Data Engineer - PlatformOne

About the company:VivSoft is an emerging technology company that specializes in using modern technologies to solve our clients' toughest mission challenges. We are focused on Cloud, Enterprise DevSecOps, Artificial Intelligence, and Digital Customer Experience to drive mission-enabling digital transformation. Our passion is building mission-focused, open, scalable solutions. We are a diverse team of strategists, engineers, designers, and creators experienced in building high-performance software and AI factory accelerators by embracing automation.
About the role: We are looking for a Data Engineer to build a real-time data processing platform that supports fast and reliable data ingestion. This role involves designing scalable systems, ensuring efficient data flow, and implementing security and compliance measures. The ideal candidate will help create a high-performance infrastructure that enables real-time access to critical data.

Key Responsibilities:

• Design and implement a scalable and maintainable real-time data processing infrastructure within our own cluster, providing full control over the environment.
• Develop and optimize data ingestion pipelines capable of efficiently handling Common Vulnerabilities and Exposures (CVE) data at scale.
• Ensure the system can serve justifications and vulnerability data in real-time, supporting critical decision-making processes.
• Work with Apache Spark, Kafka, and Iceberg to facilitate large-scale data streaming, processing, and storage.
• Architect solutions that allow seamless scaling while maintaining performance and ease of management.
• Establish and enforce robust security, access control, and governance policies to meet customer compliance and regulatory requirements.
• Continuously monitor and improve system performance, ensuring data integrity and minimizing downtime.

Skills/Qualifications:

• Security Clearance Requirement: Secret Clearance
• Strong expertise in Apache Spark and Kafka for large-scale data processing.
• Hands-on experience in building scalable data pipelines for real-time data ingestion and streaming.
• Proficiency with Iceberg (or similar alternatives) for managing large-scale data lakes and storage.
• Deep understanding of data engineering principles, distributed systems, and stream processing.
• Experience in implementing security, access control, and compliance policies in data-intensive environments.
• Ability to work with cross-functional teams and stakeholders to define, design, and implement data solutions.

• Experience in DoD or other federal cloud environments. Specifically Platform One Iron Bank.
• Experience with AI/ML Solutions