Principal Engineer - Data Ingestion & AI Pipeline
Charlotte, United States
USD 122K-173K (estimate) Senior-level Full Time
Tasks
- Balance innovation with reliability security compliance scalability and cost
- Build pipeline orchestration monitoring retry and error handling
- Create reusable patterns reference architectures standards and guardrails
- Define data quality checks completeness accuracy duplication stale content sensitive information
- Define engineering approach for program portfolio solutions
- Define enterprise technology stack
- Design incremental ingestion change detection delta processing and reprocessing
- Develop scalable data ingestion pipelines for RAG and AI
- Ensure ingestion compliance for security privacy retention and regulatory requirements
- Establish ingestion standards for lineage freshness versioning access controls auditability
- Extract transform enrich validate chunk classify and load structured and unstructured data
- Implement OCR document intelligence parsing entity extraction and content classification
- Implement transformation patterns for parsing text extraction normalization deduplication enrichment chunking classification metadata generation
- Lead planning definition and design across multiple teams
- Mentor engineers and guide technical direction
- Optimize ingested data for embeddings indexing and retrieval
- Own architecture decisions across systems and domains
- Provide technical leadership across data engineering AI platform cloud and application teams
- Provide technical oversight design reviews and code within domain
- Structure enriched data for LLM reasoning
Perks/Benefits
Skills/Tech-stack
Access Control | Apache Airflow | Apache Spark | Apache Spark SQL | Azure Data | Azure Data Factory | Azure Event | Azure Event Hubs | Azure Fabric | Azure Synapse | Azure Synapse Analytics | Batch Processing | Change detection | Chunking | Content Classification | DBT | Data Factory | Data Governance | Data Lineage | Data Quality | Data Retention | Data Security | Data orchestration | Databricks | Deduplication | Delta Processing | Document processing | ELT | ETL | Embeddings | Entity Extraction | Event Hubs | Incremental processing | Kafka | LLM | Metadata generation | Normalization | OCR | Privacy Compliance | RAG | Reprocessing | Schema inference | Semantic Search | Spark SQL | Streaming | Synapse Analytics | Text extraction | Vector Databases | Vector Search
Regions
Countries
States
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R11d ago
-
Sr. Manager Data Engineering USD 125K-188KAutomated testing | Bitbucket | CI/CD | Cloud Functions | Cloud RunSenior-level Full TimeSouthlake, TX, United States5h ago
-
Data Engineer USD 112K-160KAWS | AWS S3 | Airflow | Azure | CI/CD401k match | Family leave | Medical, dental, and vision insurance | PTO | Paid HolidaysMid-level Full TimeSanta Monica, CA, United States9h ago
-
Sr. Software Engineer, AI Infrastructure USD 144K-236KApache Beam | Apache Flink | Apache Spark | C# | C++Career development | Hybrid work | Stock grantsSenior-level Full TimeSunnyvale, CA, United States11h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | Broadcast joins | CI/CD | CachingMid-level Full TimeArlington, VA11h ago
-
Google Cloud Platform (GCP) Data Engineer USD 100K-203KApache Beam | BigQuery | CI/CD | Cloud Storage | Cloud platformSecurity clearance providedMid-level Full TimeArlington, VA11h ago
-
Lead Data Analyst, Product Analytics USD 150K-180KA/B | A/B Testing | Analytics engineering | B testing | BigQuerySenior-level Full TimeRemote, US R12h ago
-
Senior Software Engineer, MLOps USD 175K-220KAgile | Batch Processing | Consistency guarantees | Data Modeling | Data Pipelines401k match | Charitable donation match | Commuter benefits | Dental insurance | Flexible time offSenior-level Full TimeRemote - United States R13h ago
-
Senior Data Engineer USD 144K-187KAWS Lambda | Amazon DynamoDB | Amazon ECS | Amazon EKS | Amazon S3401k employer contribution | Dental insurance | Flexible Remote Friendly Work Options | Health Savings Account employer contribution | Health insuranceSenior-level Full TimeAustin, Texas14h ago
-
Senior Data Engineer - Finance USD 172K-203KARR | AWS | Apache Iceberg | Apache Spark | CRMDental insurance | Employee discounts | Flexible wellness time off | Health insurance | Hybrid workSenior-level Full TimeHybrid - San Francisco, California R14h ago
-
Machine Learning Engineer III, Core Agents USD 175K-219KAWS Bedrock | Amazon SageMaker | Embedding Models | Evaluation | IndexingSenior-level Full TimeRedwood City, CA14h ago
-
Senior-level Full TimeRemote - United States R14h ago
-
Staff Software Engineer (Data Engineer) USD 140K-165KAWS Lake | AWS Lake Formation | Amazon Athena | Apache Airflow | Apache Flink401k match | Dental insurance | Health insurance | Life insurance | Vision insuranceSenior-level Full TimeUnited States14h ago
-
Sr. Data Engineer (HYBRID) USD 145K-160KAWS | AWS Glue | AWS Lambda | Amazon Athena | Amazon DynamoDBComplimentary club membership | Hybrid work schedule | Personal training | Pilates | Shop discountsSenior-level Full TimeNew York, NY, United States R14h ago
-
Staff Data Engineer - Data Engineering USD 207K-275KAccess Control | Apache Fluss | Apache Hudi | Apache Iceberg | Apache Paimon401k matching | Dental insurance | Disability insurance | Employee stock purchase program | Flexible PTOSenior-level Full TimeLivingston, NJ / New York, NY …15h ago
-
Software Engineer USD 85K-100KBiometrics | Computer Vision | Deep learning | Facial recognition | Image Geolocation401k match | Dental insurance | Disability insurance | Health insurance | Life insuranceMid-level Full TimeSomerville, Massachusetts, United States15h ago
-
Staff Software Engineer, Data Products USD 171K-224KAWS | AWS Lambda | Amazon Athena | Amazon Redshift | Amazon S3401k retirement savings plan | Flexible time off | Generous parental leave | Health, dental, and vision insurance | Lifestyle spending account mental health support solutionsSenior-level Full TimeRemote, USA R15h ago
-
Senior Software Engineer (Data Engineer) USD 138K-162KAWS | Akka | Algorithms | Apache Spark | Cloud Native401k plan company match | Basic life insurance | Dental insurance | Health insurance | Vision insuranceSenior-level Full TimeUnited States R15h ago
-
Lead Data Modeler USD 130K-165KAWS Databricks | Azure DevOps | Business Data Analysis | Business data | Conceptual data modelingSenior-level Full TimeNew York, NY, United States15h ago
-
Senior Data Engineer - Agentic AI Engineering USD 138K-173KAWS | Access Control | Airflow | Azure | Batch dataRemote-friendlySenior-level Full TimeUnited States of America R16h ago
-
Lead Architect, GenAI Platform & Agentic AI USD 170K-220KAPI Gateway | API driven LLM | API-driven | AWS | AWS CDK401k with profit sharing | Disability coverage | Life insurance | Medical, prescription, dental and vision coverage | Paid HolidaysSenior-level Full TimeWoodbridge, NJ, United States16h ago
-
Data Engineer USD 90K-151KAPIs | Amazon Redshift | CI/CD | Cloud Computing | Cloud platform401k match | Dental insurance | Equipment provided | Health insurance | PTOMid-level Full TimeRemote, US R16h ago
-
Senior Machine Learning Engineer, Safety USD 216K-303KData Preprocessing | Deep learning | Distributed Training | Embeddings | Hugging FaceSenior-level Full TimeRemote - United States R16h ago
-
Senior-level Full TimeOrlando, FL, United States16h ago
-
Data Engineer USD 169K-175KApache Airflow | Apache Beam | Bash | BigQuery | CDC401k | Dental insurance | Health insurance | Life insurance | Paid time offMid-level Full TimeParamus, NJ, United States16h ago