Senior Data Pipeline & AI Engineer
Tasks
- Define evaluation frameworks for LLM outputs
- Design data pipelines end to end
- Develop dbt models for data transformation
- Implement pipeline observability and fault tolerance
- Integrate LLM evaluation tools with Vertex AI and Langsmith
- Investigate and resolve data pipeline production incidents
- Lead system reliability improvements and operational excellence
- Maintain and extend Go microservices for data transformation
- Optimize pipeline throughput latency and reliability
- Orchestrate data migrations across customer destinations
- Write technical design documents for architecture changes
Perks/Benefits
- Annual company meet-up
- Annual learning subsidy
- Bereavement leave
- Flexible PTO
- Learning opportunities
- Monthly productivity stipend
- Paid parental leave
- Sponsored benefits packages
- Team offsites
Skills/Tech-stack
Batch Processing | BigQuery | DBT | Data Modeling | Data pipeline | Data pipeline orchestration | Databricks | Fault Tolerance | Go | Incident Management | Java | LLM | Langsmith | Pipeline Orchestration | Pipeline observability | Python | Snowflake | Streaming processing | System Architecture | Technical documentation | Vertex AI
Education
N/A
Related jobs
-
Principal Machine Learning Engineer USD 205K-230KAWS Lambda | BigQuery | C# | CI/CD | Cloud Functions401k | Dental insurance | Health insurance | Life insurance | Paid HolidaysSenior-level Full TimeUnited States of America - Remote … R3h ago
-
Senior Software Engineer - Internal Observability USD 200K-287KAWS | Anomaly Detection | Azure | C++ | Cause analysisEntry-level Full TimeUS-CA-Menlo Park4h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Flexible remote work | Laid-back atmosphereMid-level Full TimeAtlanta, GA, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Friendly relaxed atmosphere | Opportunity to work on impactful productMid-level Full TimeAustin, TX, USA5h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeMid-level Full TimeNew York, NY, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerMid-level Full TimeBoston, MA, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Career growth | Laid-back atmosphere | Remote-friendlyMid-level Full TimePortland, OR, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Bonus opportunities | Equity opportunities | Friendly work environment | Remote-friendlyMid-level Full TimeMinneapolis, MN, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Bonus | Equity | Flexible remote setting | Friendly work environmentMid-level Full TimeTempe, AZ, USA5h ago
-
Bash | Cloud infrastructure | Data Processing | Docker | GCPAsynchronous culture | Competitive compensation not applicable | Laid-back atmosphere | Remote-friendlyMid-level Full TimeFrisco, TX, USA5h ago
-
Bash | Cloud Computing | Cloud platform | Data Processing | DockerAsynchronous culture | Friendly and laid-back atmosphere | Opportunity for career growth | Remote distributed settingMid-level Full TimeLas Vegas, NV, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous work culture | Flexible management approachMid-level Full TimeDetroit, MI, USA5h ago
-
Bash | Data Processing | Docker | GCP | Large Scale DataAsynchronous culture | Flexible management approach | Remote/distributed workMid-level Full TimeRaleigh, NC, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Flexible management approach | Remote/distributed teamMid-level Full TimeKansas City, MO, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Bonus compensation | Equity compensation | Flexible work environment | Supportive team cultureMid-level Full TimeCincinnati, OH, USA5h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Competitive compensation | Remote-friendly | Supportive teamMid-level Full TimeEvanston, IL, USA5h ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Laid-back atmosphere | Portfolio and LinkedIn submission | Remote workMid-level Full TimeRichmond, VA, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous work culture | Remote or distributed workMid-level Full TimeBakersfield, CA, USA5h ago
-
Bash | Cloud platform | Docker | GCP | Google CloudMid-level Full TimeFort Collins, CO, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | GCPAsynchronous culture | Laid-back atmosphere | Remote-friendlyMid-level Full TimeCollege Station, TX, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerMid-level Full TimeSt. Petersburg, FL, USA5h ago
-
Bash | Cloud platform | Data Ingestion | Data Processing | DockerAsynchronous culture | Career growth opportunities | Distributed work | Flexible, hands-off management | Impactful missionMid-level Full TimeTallahassee, FL, USA5h ago
-
Bash | Data Ingestion | Data Processing | Docker | GCPAsynchronous work culture | Equity | Laid-back atmosphere | Remote distributed settingMid-level Full TimeBirmingham, AL, USA5h ago
-
Bash | Cloud platform | Data Processing | Docker | GCPAsynchronous culture | Competitive benefits | Remote-friendlyMid-level Full TimeHartford, CT, USA5h ago
-
Bash | Cloud infrastructure | Cloud infrastructure as code | Cloud platform | Data ProcessingAsynchronous culture | Flexible remote work | Opportunity for impactMid-level Full TimeGreensboro, NC, USA5h ago