Lead Data Engineer
New York - 150 E 42nd, United States
USD 94K-152K Senior-level Full Time
Tasks
- Build automated validation frameworks and quality tracking pipelines
- Build data models for consumer and household identity linkage
- Build ground truth evaluation and match quality dashboards
- Continuously monitor data completeness match consistency and anomaly detection
- Create regression detection across engine versions
- Design build and maintain Snowflake pipelines for identity assets
- Design implement test and deploy via CI CD
- Document data models pipeline logic and algorithm decisions
- Enhance AI assisted matching engine feature design scoring logic and evaluation
- Implement and test matching algorithm improvements
- Investigate data anomalies and trace match decisions to source signals
- Measure match quality using precision recall and asset impact
- Perform code reviews and architectural discussions
- Support QA and on call responsibilities for production pipelines
- Translate identity requirements into reliable scalable solutions
- Write SQL and Python for transformation deduplication and enrichment
Perks/Benefits
- 401k
- Dental insurance
- Flexible paid time off
- Life insurance
- Long-term disability
- Medical insurance
- Paid Holidays
- Paid parental leave
- Paid sick leave
- Short-term disability
- Vision insurance
Skills/Tech-stack
AWS | Agile | Bash | CI/CD | Data Deduplication | Data Enrichment | Data Governance | Data Modeling | Data Quality | Data pipeline | Deterministic matching | Git | Identity matching | LLM | Machine Learning | Privacy | Probabilistic matching | Prompt engineering | Python | Record Linkage | SQL | Snowflake | Snowpark | Streams | Tasks | Unix
Education
N/A
Roles
Data Engineer | Data Engineering | Data Engineering Manager | Engineer | Engineering Manager | Lead | Lead Data | Lead Data Engineer | Manager
Regions
Countries
States
Cities
Related jobs
-
Data Engineer / BI Developer USD 91K-130KAmazon Web Services | Apache Airflow | Cloud platform | DBT | Data ModelingMid-level Full TimeCenter, Center District, IL6h ago
-
Embedded Software Engineer - Body Control Modules USD 79K-178KA/D | ASIL A D | AUTOSAR | C# | CI/CDAdoption and surrogacy expense reimbursement | Employee resource groups | Fertility treatments | Flexible family care days | Medical, dental & vision coverageSenior-level Full TimeDearborn, MI, United States7h ago
-
Software Engineer III - Senior Java Spark Developer USD 113K-188KAgile | Apache Spark | CI/CD | Concurrency | Distributed SystemsSenior-level Full TimeJersey City, New Jersey, United States8h ago
-
Data Engineer USD 97K-163KBTEQ | Data Partitioning | Database Administration | Performance Tuning | Query OptimizationSenior-level Full TimeSacramento, California, United States8h ago
-
Software Engineer, Video USD 141K-251KAV1 | AV2 | Artificial Intelligence | Audio CODEC | Automated testingMid-level Full TimeBellevue, WA | Menlo Park, CA …9h ago
-
Enterprise Risk Program Manager, AI & Data Governance USD 119K-169KAI Risk | AI risk management | Control maturity | Data Analysis | Data GovernanceMid-level Full TimeBrookfield, WI, 53005, US11h ago
-
Enterprise Risk Program Manager, AI & Data Governance USD 119K-169KControls Testing | Data Analysis | Data Governance | Data Privacy | Enterprise RiskMid-level Full TimeBrookfield, WI, 53005, US11h ago
-
Applied AI Engineer - AI Solutions USD 172K-300KAgentic Workflows | Airflow | Apache Spark | Chroma | CrewAIAnnual travel up to 25% | Employee stock options | Hybrid work | Professional developmentMid-level Full TimeNew York City, NY (Hybrid); Redwood … R16h ago
-
Manager, Enterprise Data Engineering USD 180K-247KADLS | Active Directory | Agile | Apache Spark | Automated testing401k matching | Adoption Assistance | Childcare tuition discounts | Company bonus | Fertility benefitsSenior-level Full TimeAnn Arbor, MI, United States17h ago
-
Associate Director, Data Analytics Delivery & Operations USD 215K-261KAWS | Agile | Amazon Web Services | Analytics | Business Intelligence401 K | Dental insurance | Holidays | Medical insurance | Paid time offMid-level Full TimeMettawa, IL, United States17h ago
-
AI Solutions Engineer, Talent Acquisition USD 129K-171KAPIs | Access Control | Agentic Workflows | Audit trails | AuthenticationMid-level Full TimeSeattle, Washington, United States18h ago
-
Sr. AI Engineer USD 175K-350KAgent Orchestration | Artificial Intelligence | Backend Development | Database Design | Frontend Development401k | Dental insurance | Flexible paid time off | Home office stipend | Medical insuranceSenior-level Full TimeSanta Monica, CA20h ago
-
Product Analytics Engineer USD 130K-140KA/B | A/B Testing | Airflow | B testing | DBT401k retirement savings plan | Employer-sponsored healthcare | Flexible spending account | Health savings account | Paid parental leaveSenior-level Full TimeRemote, USA R21h ago
-
Sr. Manager, Analytics & Insights USD 103K-155KAnalytics | Automation | Dashboarding | Data Governance | Data ModelingFlexible time off | Health benefits | Life insurance | Paid time off | Retirement benefitsSenior-level Full TimeFort Worth - Main, United States21h ago
-
Manager of Business Analytics - Remote USD 175K-220KA/B | A/B Testing | B testing | Business Intelligence | Cohort Analysis401k match | Career advancement opportunities | Healthcare dental vision insurance | Paid time off | Professional development opportunitiesSenior-level Full TimeGA - Remote, United States R21h ago
-
Data Engineer USD 123K-273KAWS Glue | Azure Synapse | Cloud Dataflow | DB2 | Data MiningCompany volunteering | Employee referral program | Learning opportunitiesMid-level Full TimeCIO KPop-Dallas (US152527), United States21h ago
-
Analytics Program Lead USD 189K-189KAgile | Big Data | Cloud Computing | Code Management | Data EngineeringEmployee support programs | Financial security benefits | Health and wellbeing benefits | In-office collaboration | Professional developmentSenior-level Full Time300 Kimball Drive, Parsippany, NJ, USA, …21h ago
-
Lead Data Analyst USD 107K-141KAmazon Web Services | Cloud Computing | Cloud platform | Data Architecture | Data CatalogSenior-level Full Time3965 Dallas Parkway Frisco, TX 75034, …21h ago
-
Data Engineering Manager, Commercial US USD 148K-214KAWS | Agile | Airflow | Apache Kafka | Apache SparkHealth and wellbeing benefits | Parental leave | Prevention and wellness programsMid-level Full TimeMorristown, NJ, United States21h ago
-
AI Engineer, Agentic Ad Creative (Multimodal) USD 120K-220KA/B | A/B Testing | Ad Policy Compliance | B testing | CUDA401k matching | Commuter benefits | FSA | HSA | Health, dental, and vision insuranceMid-level Full TimeMountain View, California, United States21h ago
-
Senior Manager-Data Science USD 123K-215KClickstream analytics | Clustering | Data Pipelines | Decision Trees | ExperimentationCareer development and training opportunities | Company retirement match | Confidential counseling support | Financial coaching | Flexible working modelSenior-level Full TimeNew York, NY, United States R22h ago
-
Lead Principal Machine Learning Engineer USD 169K-355KAccess Control | Agent systems | AgentOps | Auditability | Autogen401k match | Life insurance | Paid Holidays | Paid parental leave | Paid sick leaveSenior-level Full TimeSan Francisco, CA, United States22h ago
-
Continuous Deployment | Continuous integration | Dependency management | Docker | JuliaLearning and development stipend | Paid time off | Parental leave | Sick leaveSenior-level Full TimeLong Beach, California22h ago
-
Senior Data Engineer USD 126K-142KAzure Cloud | Azure Cloud Platform | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeUnited States22h ago
-
Multidisciplinary Analysis and Optimization Engineer II USD 126K-172KAerospace Engineering | Continuous Deployment | Continuous integration | Dependency management | DockerLearning and development stipend | Paid time off | Parental leave | Sick leaveEntry-level Full TimeLong Beach, California22h ago