Vice President, Data Engineering
New York, NY - PARK, United States
USD 240K-270K Executive-level Full Time
Tasks
- Build document processing pipelines from blob storage to Databricks
- Build preprocessing pipelines for NLP and LLM consumption
- Debug Spark performance issues and production incidents
- Design ADLS layouts and medallion architecture
- Design and build Spark SQL and Python ETL ELT pipelines in Databricks
- Design incremental loading with Delta Lake merge operations and CDC patterns
- Establish idempotent pipeline design with backfills and reruns
- Implement data governance with Databricks Unity Catalog
- Implement data quality frameworks schema validation and duplicate detection
- Implement text extraction using Azure AI services
- Lead postmortems and troubleshooting for production issues
- Mentor engineers through code review and design guidance
- Optimize Databricks cluster configurations for performance and cost
- Optimize Delta Lake table design partitioning and query performance
Perks/Benefits
- 401k match
- Commuter benefits
- Comprehensive medical, dental, vision
- Employee assistance program
- Flexible savings accounts
- Health savings account
- Mental health app
Skills/Tech-stack
ACID | AI Services | Active Directory | Apache Airflow | Apache Spark | Azure AI | Azure AI Services | Azure Active Directory | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure Event | Azure Event Hubs | Azure Key Vault | Azure SQL | Azure SQL Database | CDC | Change tracking | Chunking | Cognitive Search | DBT | Data Lake Storage | Data orchestration | Databricks | Databricks SQL | Databricks Workflows | Delta Lake | Delta Live Tables | Delta Live) | Dimensional Modeling | Document Intelligence | ELT | ETL | Event Hubs | Form Recognizer | Idempotency | Key Vault | NLP | NLP tokenization | OCR | Prefect | PySpark | Python | SQL | SQL Database | Schema evolution | Spark SQL | Text extraction | Unity Catalog
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Engineering Lead Analyst - VP USD 125K-188KAI Automation | AI coding | AI coding assistant | CI/CD | Cloud401k | Medical/Dental/Vision | Paid time offSenior-level Full Time6400 LAS COLINAS BLVD IRVING, United …21h ago
-
API Development | AWS | Azure | Cloud | ComplianceExecutive-level Full TimePhiladelphia, Pennsylvania, United States1d ago
-
Head of Field Engineering USD 180K-300KAgentic Systems | Agents | Applied Machine Learning | Customer Engagement | Enterprise Customer EngagementExecutive-level Full TimeSeattle, WA1d ago
-
Associate Technical Architect - DE - Databricks USD 121K-202KAWS S3 | AWS Secrets | AWS Secrets Manager | AWS Step Functions | AirflowCareer development | Innovation-focused culture | Learning and growth | Work-life balanceMid-level Full TimeUSA - Remote, United States R2d ago
-
Manager, Data Engineering USD 112K-150KApache Spark | Automated testing | Azure Cosmos | Azure Cosmos DB | Azure DataMid-level Full TimeAtlanta, Georgia3d ago
-
Manager, Data Engineering USD 160K-190KApache Spark | Automated testing | Azure Cosmos | Azure Cosmos DB | Azure DataMid-level Full TimeChicago, Illinois3d ago
-
Engineering Manager, Machine Learning Operations USD 240K-300KAgile | Amazon SageMaker | Amazon Web Services | Apache Airflow | Apache Kafka401k match | Educational stipend | Employee assistance program | Employee stock program | Paid family and paternity leaveMid-level Full TimeSeattle, Washington, United States3d ago
-
Engineering Manager, Machine Learning Operations USD 240K-300KAWS | Agile | Amazon SageMaker | Apache Airflow | Apache Kafka401k match | Annual educational stipend | Comprehensive health benefits | Employee stock program | Paid family and paternity leaveMid-level Full TimeNew York, New York, United States3d ago
-
Staff Software Engineer, Data Engineering USD 193K-253KAWS | Airflow | Amplitude | BigQuery | CI/CD401k plan | Annual cash bonus | Dental insurance | Equity grants | Flexible time offSenior-level Full TimeRemote, USA R3d ago
-
Airflow | DBT | Data Engineering | Data Governance | Data QualityMid-level Full TimePiscataway, NJ, United States3d ago
-
AWS | Agile | Airflow | Apache Spark | AzureHybrid work | Relocation supportSenior-level Full TimeDublin, OH, United States3d ago
-
Staff Engineering Analyst, AI Safety USD 189K-274KData Analysis | Data Science | Fine Tuning | Language Models | Language ProcessingSenior-level Full TimeSunnyvale, CA, USA3d ago
-
Engineering Manager, Borglet AI Infrastructure USD 207K-300KC# | C++ | Containers | Data Analysis | Distributed SystemsSenior-level Full TimeSunnyvale, CA, USA3d ago
-
Active Directory | Apache Spark | Audit Logging | Azure | Azure Active DirectoryOnsite work location CincinnatiMid-level Full TimeCincinnati, Ohio, United States3d ago
-
Engineering Technician, Energy Storage USD 135K-192KCause analysis | Data Documentation | Data logging | Electrical safety | FMEAMid-level Full TimeCarson City, NV3d ago
-
Engineering Manager, Data Platform USD 199K-275KAWS DynamoDB | AWS EMR | AWS Redshift | AWS S3 | AirflowMid-level Full TimeSan Francisco, CA, USA R3d ago
-
Engineering Manager, Financial Data Quality USD 347K-385KBilling Systems | Data Lineage | Data Modeling | Data Observability | Data QualityMid-level Full TimeSan Francisco3d ago
-
Sr Manager, Data & Analytics Engineering Services USD 125K-209KAccess Management | Azure | Cloud Computing | Cost Optimization | DatabricksSenior-level Full TimeColumbus, OH, USA - 910 John …3d ago
-
Data Engineering Tech Lead USD 106K-147KAPI Development | Agile | Apache Spark | Automated unit testing | AzureSenior-level Full TimeWORK AT HOME - OHIO, United …3d ago
-
Senior Director, Software Engineering USD 117K-163KAMQ | AWS | Agile | Amazon EMR | AngularCareer development | Flexible schedule | Hybrid workSenior-level Full TimeR24-Philadelphia, United States3d ago
-
VP, AI Software Engineering (Hybrid - Flexible Options) USD 210K-220KAPIs | Artificial Intelligence | CI/CD | Cause analysis | Cloud PlatformsPaid sick leaveExecutive-level Full TimeNew York, NY, United States3d ago
-
Sr. Director, Software Engineering (Big Data, AI/ML) USD 286K-359KAWS | Agentic AI | Agile | Anomaly Detection | Artificial IntelligenceSenior-level Full TimeChicago, IL, United States3d ago
-
AWS Glue | Amazon Athena | Amazon CloudTrail | Amazon EMR | Amazon RedshiftMid-level Full TimeEast Palo Alto, California, USA3d ago
-
Director Engineering - AI/ML USD 224K-296KAPI Design | Agentic Frameworks | Amazon Web Services | CI/CD | Cloud ComputingExecutive-level Full Time1830 Embarcadero - PALO ALTO, United …3d ago
-
Head of Data Engineering USD 300K-350KAmazon Kinesis | Apache Airflow | Apache Spark | Azure | Azure Data401k match | Commuter benefits | Comprehensive medical, dental, and vision | Disability insurance | Education sponsorship programExecutive-level Full TimeNew York, NY - PARK, United …3d ago