Principal Engineer - Data Ingestion & AI Pipeline
Charlotte, United States
USD 122K-173K (estimate) Senior-level Full Time
Tasks
- Build scalable ingestion patterns for enterprise sources
- Create reusable patterns reference architectures standards and guardrails
- Define data quality checks for completeness accuracy duplication stale content unsupported formats sensitive information
- Define engineering approach for program portfolio solutions
- Define technology tool stack within approved technologies
- Define transformation patterns for parsing text extraction normalization deduplication enrichment chunking classification metadata generation
- Design enterprise data ingestion pipelines
- Design incremental ingestion change detection delta processing and reprocessing
- Ensure ingestion compliance with security privacy retention and regulatory requirements
- Establish end to end test strategy and integration between teams
- Establish ingestion standards for lineage freshness versioning access control auditability
- Evaluate and implement OCR document intelligence parsing entity extraction classification and data quality
- Implement pipeline orchestration monitoring retry logic error handling dashboards
- Lead planning definition and design for complex features across teams
- Mentor senior engineers and influence technical direction
- Optimize ingested data for embedding indexing and retrieval for RAG
- Own architecture decisions impacting multiple teams systems or domains
- Provide technical leadership across data engineering AI platform cloud and application teams
- Provide technical oversight through design reviews and code within domain
- Structure and enrich data for effective LLM reasoning
Perks/Benefits
Skills/Tech-stack
AI Document Intelligence | Access Control | Apache Airflow | Apache Kafka | Apache Spark | Azure AI | Azure AI Document Intelligence | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Azure Fabric | Azure Synapse | Azure Synapse Analytics | Batch Processing | Change detection | Content Classification | DBT | DLP | Data Factory | Data Governance | Data Lineage | Data Quality | Data Security | Data freshness | Databricks | Delta Processing | Document Intelligence | Document parsing | ELT | ETL | Embeddings | Entity Extraction | Error Handling | Event Hubs | Incremental processing | LLM | OCR | PII Detection | Privacy Compliance | Python | RAG | Retry logic | SQL | Schema inference | Semantic Search | Streaming | Synapse Analytics | Text extraction | Vector Databases | Vector Search
Education
Bachelor of Engineering | Bachelor of Engineering in Computer Science | Bachelor of Science | Bachelor of Science in Computer Science
Roles
AI | AI Engineer | Data Engineering | Data Engineering Manager | Engineer | Engineering Manager | Manager | Principal | Principal Engineer
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R11d ago
-
Forward Deployed AI Engineer USD 180K-230KAPI | AWS | Azure | CI/CD | Cloud platform5-day workweek | Collaborative work culture | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States8h ago
-
Principal AI Engineer USD 200K-245KAgent-based | Agent-based architecture | Amazon Web Services | Azure | CI/CD5 days per week | Collaborative culture | Flexible working hours | Supportive work environmentSenior-level Full TimeNew York, New York, United States8h ago
-
Machine Learning Engineer, Sensor Pipelines USD 175K-215KC++ | Computer Vision | JAX | Machine Learning | PyTorchHybrid work scheduleMid-level Full TimeMountain View, CA, USA8h ago
-
Sr. Embedded Software/Firmware Engineer USD 180K-240KARM | ASPICE | AUTOSAR | C# | C++Competitive compensation | Dental insurance | Flexible hours | Medical insurance | Pre-IPO stock optionsSenior-level Full TimeUnited States - Remote R8h ago
-
Sr. Software Engineer, AI Infrastructure USD 144K-236KApache Beam | Apache Flink | Apache Spark | C# | C++Career development | Hybrid work | Stock grantsSenior-level Full TimeSunnyvale, CA, United States10h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | Broadcast joins | CI/CD | CachingMid-level Full TimeArlington, VA10h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | CI/CD | Cloud Storage | Cloud platformSecurity clearance providedMid-level Full TimeArlington, VA10h ago
-
AI Research Engineer Intern USD 62K-96KAutonomous Driving | Causal AI | Computer Vision | Deep learning | Edge AIHoliday pay | Medical/Dental/Vision insurance | Sick leaveEntry-level InternshipSan Jose, CALIFORNIA, United States11h ago
-
Senior ECAD Library & Engineering Data Engineer USD 160K-225K3D Component Models | APIs | Altium 365 | Altium Designer | Altium ScriptingOnsite collaborationSenior-level Full TimeSouth San Francisco, California, USA12h ago
-
Senior Data Engineer USD 144K-187KAWS Lambda | Amazon DynamoDB | Amazon ECS | Amazon EKS | Amazon S3401k employer contribution | Dental insurance | Flexible Remote Friendly Work Options | Health Savings Account employer contribution | Health insuranceSenior-level Full TimeAustin, Texas13h ago
-
Senior Data Engineer - Finance USD 172K-203KARR | AWS | Apache Iceberg | Apache Spark | CRMDental insurance | Employee discounts | Flexible wellness time off | Health insurance | Hybrid workSenior-level Full TimeHybrid - San Francisco, California R13h ago
-
Principal AI Engineer USD 130K-196KAI Safety | AWS | Agentic Workflows | Data Engineering | Evaluation401k match | Dental insurance | Disability insurance | Employee assistance program | Flexible spending accountSenior-level Full TimeDenver, CO13h ago
-
Financial Data and Analytics Manager USD 95K-142KArtificial Intelligence | Big Data | Big Data Integration | CRM | Data EngineeringHealthcare coverage | Retirement planMid-level Full TimeColumbia, Maryland, United States13h ago
-
Sr. Data Engineer (HYBRID) USD 145K-160KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon DynamoDBComplimentary club membership | Hybrid work schedule | Personal training | Pilates | Shop discountsSenior-level Full TimeNew York, NY, United States R13h ago
-
Staff Data Engineer - Data Engineering USD 207K-275KAccess Control | Apache Fluss | Apache Hudi | Apache Iceberg | Apache Paimon401k matching | Dental insurance | Disability insurance | Employee stock purchase program | Flexible PTOSenior-level Full TimeLivingston, NJ / New York, NY …14h ago
-
Cheminformatics | Cloud Computing | Data Analysis | Data Visualization | Deep learningSenior-level Full TimeRedwood City, California, United States14h ago
-
Active Learning | Bioinformatics | Chemical informatics | Cloud Computing | Data integrationSenior-level Full TimeRedwood City, California, United States14h ago
-
Senior MLOps Engineer I USD 170K-180KAWS | CI/CD | Canary deployments | Data Versioning | Deployment PipelinesAutonomy | Dental insurance | HSA Eligible | Inclusive culture | Medical insuranceSenior-level Full TimeSan Francisco, California, United States14h ago
-
Senior MLOps Engineer I USD 170K-180KAPI Integration | AWS | CI/CD | Canary Deployment | Computer VisionDental insurance | HSA options | Health insurance | Paid training | Stock options (eligibility)Senior-level Full TimeBoston, Massachusetts, United States14h ago
-
AI Developer USD 99K-198KAWS | Agentic AI | Azure | CI/CD | Data LakesHybrid work schedule | Leadership development | Professional developmentMid-level Full TimeWashington, DC14h ago
-
Software Engineer USD 85K-100KBiometrics | Computer Vision | Deep learning | Facial recognition | Image Geolocation401k match | Dental insurance | Disability insurance | Health insurance | Life insuranceMid-level Full TimeSomerville, Massachusetts, United States14h ago
-
Staff Software Engineer, Data Products USD 171K-224KAWS | AWS Lambda | Amazon Athena | Amazon Redshift | Amazon S3401k retirement savings plan | Flexible time off | Generous parental leave | Health, dental, and vision insurance | Lifestyle spending account mental health support solutionsSenior-level Full TimeRemote, USA R14h ago
-
Senior Data Engineer - Agentic AI Engineering USD 138K-173KAWS | Access Control | Airflow | Azure | Batch dataRemote-friendlySenior-level Full TimeUnited States of America R15h ago
-
Senior Machine Learning Engineer, Safety USD 216K-303KData Preprocessing | Deep learning | Distributed Training | Embeddings | Hugging FaceSenior-level Full TimeRemote - United States R15h ago