Sr Data Engineer
USA - NY - 7 Hudson Square, United States
USD 148K-199K Senior-level Full Time
Tasks
- Build pipelines using Airflow Spark and Databricks
- Collaborate with product and engineering teams to deliver data features
- Design and deploy data solutions for batch ingestion and stream processing
- Document data integrity procedures and operational processes
- Establish coding testing and deployment standards for data quality
- Evolve data models and build data pipelines for new metrics
- Investigate and resolve data pipeline failures using root cause analysis
- Maintain and update core source code datasets
- Optimize performance scalability and cost efficiency of data pipelines
- Provide technical guidance and mentorship
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Agile | Apache Airflow | Apache Spark | Bash | CICD | Data Modeling | Data Warehousing | Databricks | Databricks Delta | Delta Lake | Distributed Systems | GitHub Actions | GraphQL | Infrastructure as Code | Jenkins | Kafka | Powershell | Python | S3 | SDLC | SQL | Scala | Scrum | Snowflake | “as-code”
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Machine Learning Engineer USD 110K-150KAirflow | Bayesian Methods | CI/CD | Causal Inference | DBT401k matching | Disability coverage | Health, dental, vision insurance | Paid Holidays | Paid parental leaveMid-level Full TimeNew York, NY, US2h ago
-
Full Stack Software Engineer, Mid-Level USD 85K-110KAWS | AWS Bedrock | Agile | Amazon DocumentDB | Docker401k matching | AD and D insurance | Employee assistance program | Employee referral program | Employee resource groupsMid-level Full TimeDenver, CO, USA9h ago
-
Embedded Software QA Engineer USD 113K-149KAcceptance Testing | Agile | Automated testing | Bitbucket | BugzillaCommuter benefits | Generous time off | Healthcare benefits | Life and disability insurance | Mental health resourcesMid-level Full TimeLexington, Massachusetts, United States9h ago
-
AWS | Benchmarking | Code Quality | Code review | Data Pipelines401k matching | Dining | Flexible spending accounts | Healthcare | Onsite gymSenior-level Full TimeRemote - US R9h ago
-
Active Learning | DPO | Deep learning | Few-Shot Learning | Few-shotSenior-level Full TimeMountain View, CALIFORNIA, United States10h ago
-
Senior AWS Cloud Data Engineer USD 139K-228KAWS Database | AWS Database Migration Service | AWS Glue | AWS Lambda | AWS database migrationOnsite work locationSenior-level Full TimeAtlanta, Georgia, United States; Center Valley, …10h ago
-
Applied AI Engineer II USD 100K-295KAI Agents | AWS | Azure | CI/CD | Docker401k matching | Dental insurance | Disability insurance | Employee assistance program | Flexible spending accountSenior-level Full TimeAddison, TX (Hybrid); Durham, NC (Hybrid) R11h ago
-
Data Engineer/Software Engineer USD 70K-100KAWS | AWS Glue | Account administration | Amazon Athena | Data EngineeringMid-level Full TimeNorth Reading, MA11h ago
-
Senior Software Engineer, Data Platform USD 163K-247KAWS | Amazon EMR | Amazon Kinesis | Amazon MSK | Amazon Redshift401k match | Equity RSUs | Health insurance | Paid time offSenior-level Full TimeDenver, CO;San Francisco, CA;New York, NY;Los … R11h ago
-
Quantum Engineer USD 120K-155KCoherent control | Cold atoms | Git | Hardware Interfaces | Neutral atom401k matching | Catered team lunches | Dental insurance | Dependent care benefits | Equity optionsMid-level Full TimeBoulder, CO12h ago
-
Machine Learning Engineer (MLE) Intern USD 82K-106KAgile | CI/CD | Computer Vision | Docker | GISHybrid work locationEntry-level InternshipHerndon, VA12h ago
-
Data Engineer USD 112K-168KAWS | Apache Airflow | Azure | CI/CD | CSV401k match | Family leave | Legal insurance | Life insurance | Medical, dental, vision plansMid-level Full TimeSanta Monica, CA, United States12h ago
-
AI/ML Engineer - Vision AI USD 118K-180KCloud Computing | Computer Vision | Deep learning | Edge Computing | Image Processing401k match | Employee ownership | Health insurance | Tuition reimbursementSenior-level Full TimeRoyersford, Pennsylvania, United States12h ago
-
API Testing | CI/CD | Embedded Systems | End to End | End-to-End TestingSenior-level Full TimeBoston, MA12h ago
-
Senior Software Engineer II, Cloud Data Pipeline USD 164K-221KAPI Design | AWS | AWS CDK | AWS Step Functions | Apache AirflowHybrid work arrangementSenior-level Full TimeSeattle12h ago
-
Senior-level Full TimeRoseville, CA - HQ13h ago
-
Sr. Data Engineer II (6516) USD 152K-188KAWS | Apache NiFi | Cloudera | Data Preparation | Data labeling401k match | Dental insurance | Dependent care | Employee assistance program | Flexible work arrangementsMid-level Full TimeTysons Corner, VA13h ago
-
Analyst, Finance Analytics & AI USD 138K-180KDBT | Data Modeling | Dynamic Tables | Embeddings | GitSenior-level Full TimeUS-CA-Menlo Park13h ago
-
Senior-level Full TimeFrisco, TX13h ago
-
Analytics Engineer USD 150K-200KAccess Control | Airflow | Amazon Redshift | BigQuery | CI/CD401k | Free lunch | Healthcare Dental Vision | Snacks | Unlimited PTOMid-level Full TimeNew York Office (Flatiron)14h ago
-
AI Research Engineer USD 330K-370KBenchmarking | Constitutional AI | Data Processing | Data Quality | Data quality assessment401k match | Commuting support | Dental insurance | Equity | Fertility benefitsMid-level Full TimeSan Francisco, CA14h ago
-
Forward Deployed AI Engineer USD 150K-170KAPIs | Conversational AI | Hugging Face | LLM | Langchain401k | Community involvement | Dental insurance | Disability insurance | Flexible spending accountsMid-level Full TimePalo Alto, United States14h ago
-
Senior AI Integration Coach USD 122K-195KData Governance | Generative AI | LLM Frameworks | Language Models | Language ProcessingHybrid workSenior-level Full TimeColorado Springs, CO14h ago
-
Software Engineer II, Data Platform USD 150K-225KAWS | Apache Flink | Apache Spark | CI/CD | Data Cataloging401k matching | Dental insurance | Employee discounts | Medical insurance | Paid time offMid-level Full TimeUnited States, San Diego, CA14h ago
-
Senior Data Engineer USD 152K-207KAPI Integration | Apache Airflow | ClickHouse | Cloud Data | Cloud Data PlatformsImpact visibility | Ownership | Product collaboration | Quality focused teamSenior-level Full TimeUnited States15h ago