Senior Software Engineer, Data Processing
Tasks
- Build data quality checks
- Build end to end ingestion pipelines
- Create modality specific processing steps
- Design ingestion systems for multimodal data
- Develop parsers and validators for messy source formats
- Diagnose ingestion bottlenecks and improve performance
- Handle PHI with de identification
- Implement batch and distributed execution for unstructured data
- Improve observability and debuggability
- Optimize reliability cost and speed for high volume workloads
- Partner with product and Data Lab on new modalities
- Process and validate structured and unstructured datasets
- Support partner requirements for non standard source data
- Track data provenance metadata and usage constraints
- Use distributed parallel compute to process large workloads
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | Airflow | Batch Processing | Dagster | Data Pipelines | Data Processing | Data Quality | Data Validation | Data provenance | De-identification | Distributed Computing | Distributed execution | Observability | Python
Education
N/A
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R6d ago
-
Featured Feat. Associate Director, Data Labs USD 167K-167KAWS | Cloud Computing | Compute Infrastructure | Data Analysis | LLM GovernanceConference speaking opportunities | Hybrid work schedule | Media appearancesSenior-level Full TimeWashington, District of Columbia, 20004, United … R19d ago
-
1772 Fullstack IA engineer Senior USD 147K-178KAWS | Algorithms | Cloud Computing | Cloud platform | Code ReviewsMentorship | Remote workSenior-level Full TimeRemote R6h ago
-
Principal Software Engineer USD 105K-159KAWS | Agile | Azure | Bitbucket | CI/CDFlexible hybrid work model | Health and life insurance | Paid time off | Pension/retirement benefits | Personal/Family Care leavesSenior-level Full TimeTampa, FL, United States R7h ago
-
Principal Engineer, Technical Architect INR 2000K-5000K.NET | API first | API-first design | Access Control | Agent OrchestrationSenior-level Full TimeRemote, India R8h ago
-
Principal Engineer, Technical Architect INR 2000K-5000K.NET | API Design | Access Control | Artificial Intelligence | Batch inferenceSenior-level Full TimeRemote, India R8h ago
-
Data Engineer (Cards Team) EUR 48K-72KAWS | Amazon Redshift | Apache Airflow | Apache Flink | Apache KafkaAdditional vacation | Flexible working hours | Health days | Health insurance | Hybrid workMid-level Full TimeVilnius, Lithuania R12h ago
-
AWS | Agentic AI | Azure | DevOps | DockerCompany pension plan | Flexible working hours | Mobility budget | Remote work | Training budgetSenior-level Full TimeBerlin, Germany R12h ago
-
AWS | Agentic AI | Azure | DevOps | DockerCompany pension plan | Flexible working hours | Fully remote within Germany | Mobility budget | Remote work four days per weekSenior-level Full TimeKöln, Germany R13h ago
-
Principal AI Engineering Architect USD 224K-308KAWS | Data Processing | Docker | Google Cloud | High AvailabilityAdoption and surrogacy reimbursement | Cancer Care Program | Employee Assistance Program (EAP) | Fertility coverage | Flexible time offSenior-level Full TimeUnited States - Remote R15h ago
-
Senior-level Full TimeCA Remote R18h ago
-
Data Engineer USD 120K-150KAWS Glue | Amazon Redshift | Apache Airflow | Apache Spark | Azure Data100 percent remote | Hardware setup for home office | Paid time off | Training allowanceSenior-level Full TimeLatam R19h ago
-
Lead Data Analytics & AI Engineer EUR 78K-117KAPI | AWS | Amazon Kinesis | Apache Airflow | Apache FlinkHealth insurance | Hybrid work | Long-term illness protection | Pension | Referral bonusSenior-level Full TimeIE Dublin, Ireland R19h ago
-
MLOps Engineer EUR 30K-40KAWS | Agentic AI | Artificial Intelligence | Azure | Azure DevOpsEmployee benefits club | Remote work | Unlimited training accessSenior-level Full TimeSpain - Remote R19h ago
-
Senior-level Full TimeRemote, Canada R21h ago
-
Mid-level Full Time北京 R21h ago
-
Senior AI Solutions Engineer USD 164K-215KAI Observability | Agentic Systems | Airflow | Amazon S3 | Amazon Web Services401k plan | Annual Caltrain Pass | Fastrak reimbursement | Fitness reimbursement | Health, dental, vision insuranceSenior-level Full TimeUS - Remote R21h ago
-
Entry-level Full Time北京 R22h ago
-
Nlp / Llm 应用工程师 CNY 180K-360KAnthropic API | Chroma | Embedding | Genspark | Genspark RAG ArchitectureEntry-level Full Time北京 R22h ago
-
Mid-level Full Time北京 R22h ago
-
Entry-level Full Time北京 R22h ago
-
Senior Data Engineer USD 147K-190KAWS Lambda | Agile | Airflow | Amazon Redshift | Amazon SNS401k matching | Casual work environment | Dental insurance | Fitness reimbursement | Flexible paid time offSenior-level Full TimeRemote - US R22h ago
-
Data Visualization Engineer USD 90K-112KArcGIS | Cartography | D3 | Data Visualization | GDALCommuter benefits | Employee assistance program | Health savings account | Home office reimbursement | LinkedIn Learning accessMid-level Full TimeSan Francisco, CA R23h ago
-
Lead Data Engineer USD 160K-193KAmazon Redshift | CI/CD | DBT | ELT | ETL401k match | Adoption benefits | Employee resource groups | Flexible time off | Health insuranceSenior-level Full TimeDeerfield, IL, United States R23h ago
-
Especialista I - Technical Lead BRL 210K-252KAWS | Acceptance Testing | CI/CD | Checkmarx | Distributed SystemsSenior-level Full TimeSão Paulo, SP, Brazil R1d ago