Principal Engineer - Data Ingestion & AI Pipeline
Charlotte, United States
USD 122K-173K (estimate) Senior-level Full Time
Tasks
- Balance innovation with reliability security compliance scalability and cost
- Build pipeline orchestration monitoring retry and error handling
- Create reusable patterns reference architectures standards and guardrails
- Define data quality checks completeness accuracy duplication stale content sensitive information
- Define engineering approach for program portfolio solutions
- Define enterprise technology stack
- Design incremental ingestion change detection delta processing and reprocessing
- Develop scalable data ingestion pipelines for RAG and AI
- Ensure ingestion compliance for security privacy retention and regulatory requirements
- Establish ingestion standards for lineage freshness versioning access controls auditability
- Extract transform enrich validate chunk classify and load structured and unstructured data
- Implement OCR document intelligence parsing entity extraction and content classification
- Implement transformation patterns for parsing text extraction normalization deduplication enrichment chunking classification metadata generation
- Lead planning definition and design across multiple teams
- Mentor engineers and guide technical direction
- Optimize ingested data for embeddings indexing and retrieval
- Own architecture decisions across systems and domains
- Provide technical leadership across data engineering AI platform cloud and application teams
- Provide technical oversight design reviews and code within domain
- Structure enriched data for LLM reasoning
Perks/Benefits
Skills/Tech-stack
Access Control | Apache Airflow | Apache Spark | Apache Spark SQL | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Azure Fabric | Azure Synapse | Azure Synapse Analytics | Batch Processing | Change detection | Chunking | Content Classification | DBT | Data Factory | Data Governance | Data Lineage | Data Quality | Data Retention | Data Security | Data orchestration | Databricks | Deduplication | Delta Processing | Document processing | ELT | ETL | Embeddings | Entity Extraction | Event Hubs | Incremental processing | Kafka | LLM | Metadata generation | Normalization | OCR | Privacy Compliance | RAG | Reprocessing | Schema inference | Semantic Search | Spark SQL | Streaming | Synapse Analytics | Text extraction | Vector Databases | Vector Search
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R11d ago
-
Lead Architect, GenAI Platform & Agentic AI USD 170K-220KAPI Gateway | API driven LLM | API-driven | AWS | AWS CDK401k with profit sharing | Disability coverage | Life insurance | Medical, prescription, dental and vision coverage | Paid HolidaysSenior-level Full TimeWoodbridge, NJ, United States9h ago
-
Senior-level Full TimeOrlando, FL, United States9h ago
-
Solutions Engineer USD 95K-130KAgile | Audit controls | Bug Fixing | Dashboards | Data LineageOnsite positionMid-level Full TimeNew Albany, OH, United States12h ago
-
Reindeer- Software Engineer (Infrastructure) USD 180K-250KAPI Integration | AWS | Authentication | Authorization | AzureSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL13h ago
-
C++ | DPO | GNN | Graph Neural Networks | Heterogeneous GraphsEntry-level Full TimeSeattle, Washington, United States14h ago
-
Forward Deployed Engineer IV, Applied AI, Google Cloud USD 207K-301KAPI Integration | Agent systems | Cloud platform | Conversational AI | DebuggingTravel up to 50%Senior-level Full TimeSan Francisco, CA, USA; Austin, TX, …15h ago
-
Senior Software Engineer, AI/ML, Search Ecosystems USD 174K-253KAlgorithms | C++ | Data Processing | Data Structures | Data debuggingSenior-level Full TimeMountain View, CA, USA15h ago
-
Senior Lead Data Architect USD 170K-205KAI-assisted validation | Access Controls | Agent Toolchains | Application development | Artificial IntelligenceBackup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimePlano, TX, United States19h ago
-
Data Analytics Architect USD 136K-220KArtificial Intelligence | Business Intelligence | Data Engineering | Data Governance | Data LakeHybrid work | One day per week remoteSenior-level Full TimeCenter, Center District, IL20h ago
-
Analytics Engineer USD 118K-177KAWS | Airflow | Apache Spark | Census | DBT401k match | Dental insurance | Health insurance | Life insurance | Tuition reimbursementSenior-level Full TimeNewton Office , United States1d ago
-
Senior Director, Data Engineering USD 182K-291KAWS | Agile | CI/CD | Change Data Capture | ContainerizationSenior-level Full TimeNewton Office , United States1d ago
-
Senior Data Engineer - AVP USD 120K-175KAML | AWS | AWS Glue | AWS Lambda | Amazon EC2Medical Dental Vision 401k Life Insurance Paid LeaveSenior-level Full TimeBuilding 400-Whippany Campus, Jefferson Park, United …1d ago
-
Senior Healthcare Data Analytics Lead - Remote USD 91K-163KBusiness Intelligence | Claims data | Dashboards | Data Modeling | Data NormalizationRemote workSenior-level Full TimeNewton Center, Massachusetts, UNITED STATES; REMOTE R1d ago
-
AI Agents | API Integration | Cloud Native | Cloud services | Cloud-native development401k plan | Adoption reimbursement | Disability benefits | Employee assistance program | Employee discountsSenior-level Full TimeBothell, Washington1d ago
-
AI Agents | API Integration | Cloud Native | Cloud services | Cloud-native development401k plan | Adoption reimbursement | Disability benefits | Employee assistance programs | Employee discountsSenior-level Full TimePlano, Texas1d ago
-
AI Agents | AI Governance | API Integration | Backend APIs | Cloud services401k plan | Adoption reimbursement | Disability benefits | Employee assistance programs | Employee discountsSenior-level Full TimeSan Ramon, California1d ago
-
Lead Applied Scientist, Search & Information Retrieval USD 147K-274KAWS SageMaker | AzureML | Distributed Systems | Elasticsearch | Hugging FaceCareer development and growth | Commuter benefits | Employee assistance program | Fitness reimbursement | Flexible work-life balanceSenior-level Full TimeUnited States of America, New York, … R1d ago
-
Big Data/PySpark Engineering Lead - Vice President USD 113K-170KApache Spark | Automated testing | Avro | Bitbucket | CI/CD401k matching | Dental insurance | Disability insurance | Health insurance | Life insuranceSenior-level Full TimeLocation(s): Tampa, Florida, United States1d ago
-
Senior Data Engineer USD 129K-165KAWS | Apache Airflow | BigQuery | CI/CD | ChatGPT401k matching | Healthcare packages | Hybrid work schedule | Online learning platform with guided career tracks | Paid time offSenior-level Full TimeUSA - Georgia - Alpharetta - …1d ago
-
Manager Data Engineering - Agentic AI USD 115K-192KCanonical Data | Canonical Data Modeling | Data Enrichment | Data Ingestion | Data ModelingFlexible working hours | Sabbatical | Shared parental leave | Study assistance | Wellbeing initiativesMid-level Full TimeHome based-North Carolina, United States R1d ago
-
EDO Data Governance and Architecture Lead USD 133K-156KAWS | Access Management | Apache Spark | Azure | CollibraDental insurance | Disability insurance | Flexible schedule | Healthcare benefits | Hybrid work 3 or more days per weekSenior-level Full TimeMinneapolis, MN, United States1d ago
-
Principal Scientist, DSCS Digital Technologies–Laboratory Automation, Time Series Data Strategy (Hybrid) USD 173K-272KAPI Integration | Analytical technology | Cloud Data | Cloud Data Platforms | Continuous DataSenior-level Full TimeUSA - New Jersey - Rahway, … R1d ago
-
Artificial Intelligence (AI) Engineer USD 120K-170KAI Agents | APIs | C# | Cloud services | Data ArchitectureHybrid work | Professional development | Remote work optionMid-level Full TimeHouston, TX, USA1d ago
-
Sr. Data Engineer - Reston, VA - Freewheel USD 129K-203KApache Spark | Cloud Data | Cloud Data Warehouse | Data Lineage | Data ModelingSenior-level Full TimeVA - Reston, 11951 Freedom Dr …1d ago