Lead Data Engineer
New York - 150 E 42nd, United States
USD 94K-152K Senior-level Full Time
Tasks
- Build automated validation frameworks and quality tracking pipelines
- Build data models for consumer and household identity linkage
- Build ground truth evaluation and match quality dashboards
- Continuously monitor data completeness match consistency and anomaly detection
- Create regression detection across engine versions
- Design build and maintain Snowflake pipelines for identity assets
- Design implement test and deploy via CI CD
- Document data models pipeline logic and algorithm decisions
- Enhance AI assisted matching engine feature design scoring logic and evaluation
- Implement and test matching algorithm improvements
- Investigate data anomalies and trace match decisions to source signals
- Measure match quality using precision recall and asset impact
- Perform code reviews and architectural discussions
- Support QA and on call responsibilities for production pipelines
- Translate identity requirements into reliable scalable solutions
- Write SQL and Python for transformation deduplication and enrichment
Perks/Benefits
- 401k
- Dental insurance
- Flexible paid time off
- Life insurance
- Long-term disability
- Medical insurance
- Paid Holidays
- Paid parental leave
- Paid sick leave
- Short-term disability
- Vision insurance
Skills/Tech-stack
AWS | Agile | Bash | CI/CD | Data Deduplication | Data Enrichment | Data Governance | Data Modeling | Data Quality | Data pipeline | Deterministic matching | Git | Identity matching | LLM | Machine Learning | Privacy | Probabilistic matching | Prompt engineering | Python | Record Linkage | SQL | Snowflake | Snowpark | Streams | Tasks | Unix
Education
N/A
Roles
Data Engineer | Data Engineering | Data Engineering Manager | Engineer | Engineering Manager | Lead | Lead Data | Lead Data Engineer | Manager
Regions
Countries
States
Cities
Related jobs
-
Data Analytics Engineer USD 139K-230KAWS | Azure | Azure Data | Azure Data Lake | Cloud platformDEI initiatives | Medical/Dental/Vision | Mental health support | Retirement plans | Rewards and recognitionSenior-level Full TimeRemote, United States1h ago
-
Data Engineer USD 126K-208KAPI Integration | AWS | Airflow | Azure | BigQueryDiversity, Equity, and Inclusion Initiatives | Employee rewards and recognition | Mental health support | Sustainability initiatives | Wellness programMid-level Full TimeRemote, United States1h ago
-
AWS | AWS Glue | AWS Lambda | Amazon EMR | Amazon RedshiftCollaborative environment | Mentorship and growth | Mission-driven impactEntry-level Full TimeVaughan, Ontario; Remote, Canada; Remote, US … R2h ago
-
Generative AI Engineer USD 100K-170KAPIs | AWS Bedrock | AWS Glue | AWS Lambda | AWS Step FunctionsSenior-level Full TimeCenter, Center District, IL4h ago
-
Senior-level Full TimeAnnapolis Junction, United States R4h ago
-
Mid-level Full TimeBedminster, United States4h ago
-
Software Engineer, Parallel Scientific Computing USD 130K-190KBenchmarking | C++ | CUDA C++ | Computer Architecture | High PerformanceEntry-level Full TimeRedwood City4h ago
-
Software Engineer, Scientific Computing USD 130K-190KCUDA | Inverse problems | Numerical Methods | Performance optimization | PythonEntry-level Full TimeRedwood City4h ago
-
AWS | As-a-Service | BGP | CASB | Cloud PlatformsMid-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL4h ago
-
Entry-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL4h ago
-
Product Manager, Alternative Data - Analytics USD 140K-295KAgile | Alternative data | Analytics Metrics | Data analytics | Financial Services401k match | Dental insurance | Life insurance | Medical insurance | Paid HolidaysMid-level Full TimeNew York5h ago
-
Palantir Engineer - AI and Data Science Engineer III USD 131K-218KArtificial Intelligence | Data Engineering | Data Modeling | Data Pipelines | Data TransformationActive secret clearanceSenior-level Full TimeHuntsville, Alabama, United States5h ago
-
Access Controls | Artificial Intelligence | Dashboards | Data Auditing | Data CatalogingSenior-level Full TimeCharlotte, North Carolina, United States; Jacksonville, …5h ago
-
.NET | AI Agent | AI agent orchestration | API Gateway | AWSMid-level Full TimeDallas, Texas, United States; Hermitage, Tennessee, …5h ago
-
Data Engineer III USD 124K-207KAmazon Web Services | Apache Spark | Cloud platform | Databricks | Google CloudAbility to obtain security clearance | Professional developmentSenior-level Full TimeRosslyn, Virginia, United States5h ago
-
Senior Data Engineer USD 179K-318KAI/ML | Apache Airflow | Big Data | Data Deduplication | Data MatchingHealth insurance | Holiday pay | Learning and development | Life insurance | Long-term disabilitySenior-level Full TimeUSA-VA-Chantilly6h ago
-
Agent Evaluation & Evolution Machine Learning Engineer Intern (AML-Ark-US) - 2027 Summer USD 162K-316KAutomation | Benchmarking | Data Analysis | Language Models | Language ProcessingCareer development workshops | Hands-on projects | Industry exposure | Social eventsEntry-level InternshipSan Jose, California, United States6h ago
-
Visual Generation & Multimodal Evaluation Machine Learning Engineer Graduate (AML-Ark-US) - 2027 Start USD 128K-256KAutomated metrics | Computer Vision | Data Engineering | Data Pipelines | Deep learningEntry-level Full TimeSan Jose, California, United States6h ago
-
Data Lake Infrastructure & Data Analytics Research Engineer Graduate (AML-Ark-US) - 2027 Start USD 128K-256KAnalytics | Batch Processing | Data Engineering | Data Governance | Data LakeEntry-level Full TimeSan Jose, California, United States6h ago
-
Data Lake Infrastructure & Data Analytics Research Engineer Graduate (AML-Ark-US) - 2027 Start (PhD) USD 162K-316KBatch Processing | Data Engineering | Data Governance | Data Lake | Data QualityEntry-level Full TimeSan Jose, California, United States6h ago
-
Data Lake Infrastructure & Data Analytics Research Engineer Intern (AML-Ark-US) - 2027 Summer USD 128K-256KBatch Processing | Cloud Computing | Data Engineering | Data Governance | Data LakeDevelopment workshops | Industry exposure | Internship experience | Social eventsEntry-level InternshipSan Jose, California, United States6h ago
-
Autoscaling | Container Runtime | Disaster Recovery | Docker | ElasticityEntry-level Full TimeSeattle, Washington, United States6h ago
-
Agent Evaluation & Evolution Machine Learning Engineer Graduate (AML-Ark-US) - 2027 Start USD 121K-243KBenchmarking | Data Pipelines | Fine Tuning | Human evaluation | LLM EvaluationEntry-level Full TimeSeattle, Washington, United States6h ago
-
Agent Evaluation & Evolution Machine Learning Engineer Intern (AML-Ark-US) - 2027 Summer USD 162K-316KBenchmarking | Data Analysis | Deep learning | Language Models | Language ProcessingDevelopment workshops | Industry exposure | Learning programs | Social eventsEntry-level InternshipSeattle, Washington, United States6h ago
-
Visual Generation & Multimodal Evaluation Machine Learning Engineer Graduate (AML-Ark-US) - 2027 Start USD 121K-243KComputer Vision | Data pipeline | Deep learning | Fine Tuning | Language ModelsEntry-level Full TimeSeattle, Washington, United States6h ago