AI Pipeline Engineer
Tasks
- Build high throughput data loading for training
- Build ingestion systems for multimodal data
- Construct evaluation datasets with integrity controls
- Design data pipelines for AI training and evaluation
- Design storage architectures for cost throughput and latency
- Develop dataset versioning and lineage
- Document data systems schemas and procedures
- Drive observability for data quality and pipeline health
- Implement data cleaning and quality assurance
- Implement data privacy redaction and consent enforcement
- Implement labeling workflows and active learning
- Optimize cost and performance with compression and caching
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Beam | CI/CD | Data Lineage | Data Modeling | Data Quality | Data loading | Data quality assurance | Dataset versioning | Distributed Systems | GPU Utilization | High Throughput | High Throughput Data Loading | High-throughput data | Java | Python | Quality Assurance | Ray | Scala | Spark | Testing
Education
Roles
AI | AI Pipeline Engineer | Data Engineer | Engineer | Pipeline Engineer
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R26d ago
-
Senior Data Engineer (Remote) USD 108K-150KData Architecture | Data Lake | Data Modeling | Data Quality | Data VisualizationRemote workSenior-level Full TimeUnited States R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache Kafka | Apache NiFiHybrid work model | Remote work | US government clearanceSenior-level Full TimeAnnapolis Junction, MD; San Antonio, TX R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache NiFi | Data EngineeringHybrid work model | Onsite Days Per Week | Remote work option | US government clearance requirementsSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data IngestionHybrid work schedule | Onsite work 1 to 2 days per week | Remote work 3 to 4 days per weekSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data pipelineHybrid workSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache NiFi | Data PipelinesHybrid schedule | US government clearance supportSenior-level Full TimeHerndon, VA; San Antonio, TX R2h ago
-
Amazon S3 | Amazon SQS | Apache NiFi | Data pipeline | ETLHybrid work model | Onsite work 1 to 2 days per week | Remote workSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache NiFi | Data PipelinesHybrid work model | Remote work flexibilitySenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Data Engineering | ETL | Integration TestingHybrid work model | Onsite Days Per Week | Remote workSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache Kafka | Apache NiFiSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data PipelinesCoding challenge during interview process | Hybrid work model | US federal security clearance supportSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache NiFi | Data Ingestion | Data pipelineHybrid work model | US government clearance sponsorshipSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
AWS S3 | AWS SQS | Data Pipelines | ETL | JavaHybrid work model | Onsite 1 to 2 days per week | Remote work option | US government clearance supportSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Apache NiFi | ETL | Integration TestingHybrid work | US government security clearance supportSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data IngestionCoding challenge interview process | Hybrid work | US government security clearance processingSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Apache NiFi | Data Pipelines | ETLHybrid work model | Onsite 1 to 2 days per week | Remote work daysSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data IngestionHybrid work | Onsite 1 to 2 days per week | Remote workSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data IngestionHybrid work model | Onsite 1 to 2 days per week | Remote work | Secret clearance supportSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache Kafka | Apache NiFiActive secret clearance | Hybrid work model | Top Secret clearance requirementSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache NiFi | Data IngestionHybrid work model | Onsite 1 to 2 days per week | Remote workSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon SQS | Amazon Web Services | Apache NiFi | Apache S3 | Data PipelinesHybrid work | Onsite days 1 to 2 per week | Remote work | US government clearance supportSenior-level Full TimeAnnapolis Junction, MD R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | ETLHybrid workSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Amazon Web Services | Apache Kafka | Apache NiFiGovernment clearance support | Hybrid work model | US citizen requirementsSenior-level Full TimeHerndon, VA R2h ago
-
Amazon S3 | Amazon SQS | Apache Kafka | Apache NiFi | Data IngestionCoding challenge interview process | Hybrid work model | Secret clearance required | TSSCI clearance requiredSenior-level Full TimeHerndon, VA R2h ago