Principal Engineer - Data Ingestion & AI Pipeline
Charlotte, United States
USD 122K-173K (estimate) Senior-level Full Time
Tasks
- Build scalable ingestion patterns for enterprise sources
- Create reusable patterns reference architectures standards and guardrails
- Define data quality checks for completeness accuracy duplication stale content unsupported formats sensitive information
- Define engineering approach for program portfolio solutions
- Define technology tool stack within approved technologies
- Define transformation patterns for parsing text extraction normalization deduplication enrichment chunking classification metadata generation
- Design enterprise data ingestion pipelines
- Design incremental ingestion change detection delta processing and reprocessing
- Ensure ingestion compliance with security privacy retention and regulatory requirements
- Establish end to end test strategy and integration between teams
- Establish ingestion standards for lineage freshness versioning access control auditability
- Evaluate and implement OCR document intelligence parsing entity extraction classification and data quality
- Implement pipeline orchestration monitoring retry logic error handling dashboards
- Lead planning definition and design for complex features across teams
- Mentor senior engineers and influence technical direction
- Optimize ingested data for embedding indexing and retrieval for RAG
- Own architecture decisions impacting multiple teams systems or domains
- Provide technical leadership across data engineering AI platform cloud and application teams
- Provide technical oversight through design reviews and code within domain
- Structure and enrich data for effective LLM reasoning
Perks/Benefits
Skills/Tech-stack
AI Document Intelligence | Access Control | Apache Airflow | Apache Kafka | Apache Spark | Azure AI | Azure AI Document Intelligence | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Azure Fabric | Azure Synapse | Azure Synapse Analytics | Batch Processing | Change detection | Content Classification | DBT | DLP | Data Factory | Data Governance | Data Lineage | Data Quality | Data Security | Data freshness | Databricks | Delta Processing | Document Intelligence | Document parsing | ELT | ETL | Embeddings | Entity Extraction | Error Handling | Event Hubs | Incremental processing | LLM | OCR | PII Detection | Privacy Compliance | Python | RAG | Retry logic | SQL | Schema inference | Semantic Search | Streaming | Synapse Analytics | Text extraction | Vector Databases | Vector Search
Education
Bachelor of Engineering | Bachelor of Engineering in Computer Science | Bachelor of Science | Bachelor of Science in Computer Science
Roles
AI | AI Engineer | Data Engineering | Data Engineering Manager | Engineer | Engineering Manager | Manager | Principal | Principal Engineer
Regions
Countries
States
Related jobs
-
AI Developer and Data Lead USD 91K-184K.NET | AI Security | API Development | AWS Lambda | Amazon AuroraSenior-level Full TimeWashington, DC8h ago
-
Sr. Application Software Engineer, Data Analytics USD 160K-225KAngular | C# | CI/CD | Computer Vision | Continuous integrationExtended hours | Travel | Weekend workSenior-level Full TimeBastrop, TX10h ago
-
Senior-level Full TimeOnsite - Austin, TX11h ago
-
AWS Bedrock | Agent systems | Anthropic API | Autogen | Azure401k matching program | Adoption Assistance | Development and career growth opportunities | Fertility treatments | Flexible work schedulesSenior-level Contract Full TimeRemote, OR, United States R12h ago
-
Senior Data Engineer USD 90K-110KAWS | Agile | Apache NiFi | Data Architecture | Data ModelingAutonomy | Flexible working hours | Global employee assistance programme | Online training videos | Teambuilding eventsSenior-level Full TimeNew York, United States12h ago
-
AI Engineer 1 - Compliance, Quality & Testing USD 80K-150KAI-assisted testing | CI/CD | Code review | Cypress | Defect Tracking401k match | Dental insurance | Holidays | Medical insurance | Paid time offEntry-level Full TimeWashington, DC14h ago
-
Data Engineer USD 74K-133KAgile | Apache Airflow | BigQuery | Cloud Composer | Cloud Data401k retirement plan | Dental insurance | Disability insurance | Flexible time off | Health insuranceMid-level Full TimeLisle, IL, United States R14h ago
-
API Testing | Cypher | Data Quality | DataOps | DevOpsBenefits | Competitive pay | Growth opportunity | Remote work | Travel requiredSenior-level Full TimeReston, VA, United States R16h ago
-
Sr. Manager, Digital & AI Success USD 215K-338KArtificial Intelligence | Business Requirements | Cloud Computing | Customer Analytics | Data WorkflowEmployee stock purchase plan | Health insurance | Leave options | Life insurance | Paid time offSenior-level Full TimeAtlanta, GA Office (ATLANTA)16h ago
-
Principal AI/ML Scientist USD 150K-207KAWS | AWS GovCloud | Artificial Intelligence | Azure | Azure AIPublic trust suitabilitySenior-level Full TimeARLINGTON, VA, United States16h ago
-
Principal AI Program Manager USD 145K-193KAI Governance | Artificial Intelligence | Change Management | Data Preparation | Enterprise Architecture401k matching | Paid sick leave | Paid vacation time | Tuition reimbursementSenior-level Full TimeSan Jose, CA, United States17h ago
-
Principal Engineer - Data Platform USD 221K-387KAWS | Airflow | Apache Hive | Apache Iceberg | Apache ImpalaRemote workSenior-level Full TimeSanta Clara, California, United States R17h ago
-
AI Engineer 1 - Platform Integration & AI/Data USD 80K-150KAWS | Agentic Systems | Backend Development | Backend Services | CI/CD401k match | Dental insurance | Holidays | Medical insurance | Paid time offEntry-level Full TimeWashington, DC17h ago
-
IT Infrastructure Engineer II USD 139K-178KAWS Organizations | Access Management | Amazon Web Services | Ansible | Application FirewallHealth and wellness resources | Remote work | Wellness FridaysSenior-level Full TimeRemote - United States R17h ago
-
Product Manager, Data Analytics and Cyber Resilience USD 149K-224KAnalytics | Automation | Cloud technologies | Cybersecurity | Data ArchitectureCompany-sponsored team events | Flexible time off | In-office work at Santa Clara office | Wellness resourcesMid-level Full TimeSanta Clara, California17h ago
-
Agile | Automated testing | CI/CD | Cloud Computing | CrewAIDental insurance | Health insurance | Vision insuranceMid-level Full TimeAshburn, VA, United States18h ago
-
Senior Data Engineer - Databricks USD 180K-248KAWS | Access Control | Amazon Web Services | Apache Spark | Automated testing401k match | Corporate Benefit Program | Discounted pet insurance | Educational resources | Employee Referral Bonus ProgramSenior-level Full TimeUS - Remote R19h ago
-
AI Machine Learning Skill 2-FFPP-8904 USD 78K-250KC# | Data Governance | Data Modeling | Data pipeline | Java401k plan with company match | Dental insurance | Diverse inclusive workplace | Employee referral programs | Flexible spending accountsMid-level Full TimeHanover, MD20h ago
-
AWS | AWS SageMaker | Azure | Cloud Pak for Data | Cloud infrastructureAccess to national security mission work | Hybrid work | Travel opportunitiesSenior-level Full TimeUSA-VA-Herndon20h ago
-
AI-assisted software development | AWS | Agentic AI | Azure | Cloud ComputingSenior-level Full TimeUSA-VA-Herndon20h ago
-
Analytics Engineer USD 115K-150KAgile | Azure DevOps | CI/CD | DBT | Data GovernanceAdoption Assistance | Dental insurance | Disability insurance | Educational assistance | Flexible spending accountMid-level Full TimeHouston, Texas | Tulsa, Oklahoma | …21h ago
-
AI Engineer USD 180KAgent Orchestration | Cost Management | Data Pipelines | Distributed Systems | LLM401k | Commuter benefits | Dental insurance | Flexible spending | Health insuranceMid-level Full TimeNew York, New York, United States …21h ago
-
Data & Analytics Specialist USD 87K-135KAPI Integration | Alteryx | DAX | JavaScript | Power AppsAdoption Assistance | Educational assistance | Flexible spending account | Health savings account | Life insuranceMid-level Full TimeWichita, Kansas21h ago
-
Data Platform & Engineering Specialist USD 100K-130KAWS | Amazon Kinesis | Azure | Azure Event | Azure Event HubsDental insurance | Educational assistance | Flexible spending accounts | Health insurance | Health savings accountsMid-level Full TimeLincoln, Nebraska21h ago
-
Machine Learning Leader - Optical Solutions USD 180K-300KAnomaly Detection | Data analytics | Image Processing | Java | Machine LearningAdoption Assistance | Disability insurance | Educational assistance | Flexible spending account | Health savings accountSenior-level Full TimeFremont, California21h ago