Senior Software Engineer, Data Processing
Tasks
- Build data quality checks
- Build end to end ingestion pipelines
- Create modality specific processing steps
- Design ingestion systems for multimodal data
- Develop parsers and validators for messy source formats
- Diagnose ingestion bottlenecks and improve performance
- Handle PHI with de identification
- Implement batch and distributed execution for unstructured data
- Improve observability and debuggability
- Optimize reliability cost and speed for high volume workloads
- Partner with product and Data Lab on new modalities
- Process and validate structured and unstructured datasets
- Support partner requirements for non standard source data
- Track data provenance metadata and usage constraints
- Use distributed parallel compute to process large workloads
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Airflow | Batch Processing | Dagster | Data Pipelines | Data Processing | Data Quality | Data Validation | Data provenance | De-identification | Distributed Computing | Distributed execution | Observability | Python
Education
N/A
Related jobs
-
MLOps Engineering Specialist GBP 55K-58KAWS | AWS CDK | AWS CloudFormation | AWS Glue | AlertingDiscounted mobile and broadband | Gym membership discounts | Holiday purchase scheme | Online GP service | Paid Maternity LeaveMid-level Full TimeLondon, GB, E1 8EP R4h ago
-
Senior ML/AI Modeler, Risk Automation Machine Learning USD 160K-283KAPI Integration | AWS | Airflow | Artificial Intelligence | Batch ProcessingFlexible time off | Medical insurance | Modern family planning | Remote work | Retirement savings plansSenior-level Full TimeSeattle, WA, United States of America R7h ago
-
CI/CD | Code review | Data Modeling | Database Design | DockerFlexible work across Brazil | Fully remote | Growth potentialSenior-level Full TimeBrazil R7h ago
-
API Design | AWS | Automation Anywhere | Blue Prism | CrewAIDental insurance | Fully remote | Gym membership | Health insurance | Home-office allowanceSenior-level Full TimeBrazil R8h ago
-
Autonomy | C++ | DDS | Edge Computing | Embedded Systems401k | Health insurance | Paid time offMid-level Full TimeSan Carlos - Hybrid R11h ago
-
Autonomy | C++ | DDS | Edge Computing | Embedded Systems401k plan | Health insurance | Paid Company Holidays | Paid time off | Phone stipendSenior-level Full TimeSan Carlos - Hybrid R11h ago
-
AWS | AWS Glue | AWS Step Functions | Agile | Amazon EMRFlexible start and finish times | Flexible working hours | Hybrid work model | Job-sharing | Part-time arrangementsSenior-level Contract Full Time TemporarySydney, NSW - CBP North, 1 … R13h ago
-
Alteryx | Azure Data | Azure Data Factory | CI/CD | CloverDXMid-level Full TimeMakati City, Metro Manila, Philippines R13h ago
-
Alteryx | Azure Data Factory | CI/CD | CloverDX | Data ComplianceMid-level Full TimeMakati City, Metro Manila, Philippines R13h ago
-
Alteryx | Amazon Web Services | Azure | CSV | Cloud StorageCollaboration culture | Continuous improvement | Hybrid work environmentSenior-level Full TimeMakati City, Metro Manila, Philippines R13h ago
-
Agent Orchestration | Context engineering | Evaluation | FastAPI | KubernetesExtra vacation days | Health insurance | High-spec hardware | Hybrid work options | Pension benefitsSenior-level Full TimeVilnius, Vilnius City Municipality, Lithuania R13h ago
-
Senior ML Engineer USD 155KAWS Lambda | Batching | BentoML | Caching | Distributed tracingHome office setup | National holidays | Paid time off | Remote flexibility | Stock optionsSenior-level Full TimeBrazil R14h ago
-
Staff Data Engineer, Ads USD 248K-279KAPI Integration | Airflow | Alerting | Anomaly Detection | BigQuerySenior-level Full TimeRemote (U.S.) R14h ago
-
Senior Solution Engineer USD 165K-216KCloud Computing | Data Lake | Data Warehouse | Data fabric | Data mesh401k matching | Flexible PTO | Health, dental, vision coverage | Professional development budgetSenior-level Full TimeUS-MN-Remote R15h ago
-
Sr. AI Engineer GBP 90K-120KAPI Integration | Agentic Workflows | Language Models | Large Language Models | Machine Learning401k match | Charitable giving program | Dental insurance | FSA | HSASenior-level Full TimeUK - Remote R15h ago
-
Machine Learning Engineer, Conversion ML USD 207K-275KBatch Processing | Deep learning | Machine Learning | Machine Learning Pipelines | Model DriftDental insurance | Equity | Health insurance | Remote work | Vision insuranceSenior-level Full TimeUnited States (Remote) R15h ago
-
Access Control | Apache Spark | Azure Data | Azure Data Factory | Azure Data LakeSenior-level Full TimeRemote R16h ago
-
Applied AI Engineer USD 205K-282KAMS Integration | API Integration | Carrier Portal | Carrier Portal Integration | Data integration401k plan | FSA | Flexible vacation | HSA | Health insuranceExecutive-level Full TimeAustin - HQ R16h ago
-
Distinguished AI Engineer USD 350K-600KAgent systems | DBT | EHR Integration | Fine Tuning | GuardrailsSenior-level Full TimeNew York, NY R16h ago
-
Distinguished AI Engineer USD 350K-600KAgent systems | DBT | Data integration | Deep learning | EHR IntegrationHybrid workSenior-level Full TimeNew York, NY R16h ago
-
MLOps / Machine Learning Engineer USD 180K-234KCloud Computing | Data Pipelines | Deep learning | Deep reinforcement learning | Distributed TrainingFlexible work arrangements | Professional development | Remote workSenior-level Full TimeRemote job R17h ago
-
A/B | A/B Testing | AWS | Alerting | B testingAnnual refresh grants | Equity grant | Flex First Work Options | On-call rotation | Remote workSenior-level Full TimeUnited States - Remote R17h ago
-
Senior Data Analytics Engineer USD 100K-140KAWS | Apache Airflow | Azure | BigQuery | DBT401k match | Flexible work arrangements | Health insurance | Life insurance | Paid time offSenior-level Full TimeRemote - US R18h ago
-
Senior Data Engineer USD 100K-140KAmazon Web Services | Apache Airflow | Cloud platform | Data Modeling | Data Validation401k match | Flexible benefits | Health insurance | Life insurance | Lifestyle spending accountSenior-level Full TimeRemote - US R18h ago
-
Senior Staff Machine Learning Engineer, Notifications USD 266K-372KAgentic AI | Data Pipelines | Generative AI | Go | LLM integration401k employer match | Caregiving support | Family planning support | Flexible vacation | Gender-affirming careSenior-level Full TimeRemote - United States R19h ago