数据管线高级工程师
Tasks
- Build data management platform from ingestion to training
- Build data visualization
- Build distributed high throughput low latency systems
- Collaborate with model teams to deliver business requirements
- Design and build data pipeline core workflow
- Design data models
- Develop data access and collaboration workflows
- Develop data annotation quality checks tools
- Develop data cleaning tools
- Develop data mining toolchain
- Enable fast data retrieval
- Implement data lineage tracking
- Implement data version control
- Implement metadata management
- Manage memory and I O optimization
- Optimize distributed data processing performance
- Perform data cleaning and standardization
- Perform data collection and synchronization
- Perform offline data processing
- Perform real-time data processing
- Service data APIs
- Solve large scale data transfer bottlenecks
- Support data collection and logging
- Support unified data ingestion across teams
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Data Lineage | Data Processing | Data Versioning | Distributed Systems | Docker | ETL | Go | Java | Kafka | Kubernetes | Lance | Metadata Management | MongoDB | MySQL | PostgreSQL | Pulsar | Python | RabbitMQ | Real Time | Real-time Data | Real-time Data Processing | Redis | SQL
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
Senior-level Full Time InternshipGuangzhou1h ago
-
AWS | Azure | Cloud Computing | Data Preprocessing | Entity recognitionAccident insurance | Annual leave | Dental coverage | Employee discount | Life insuranceSenior-level Full TimeHong Kong, Hong Kong, China13h ago
-
AI GPU Arch Perf Optimization Intern CNY 38K-50KAttention | CUDA | GEMM | OpenCL | Operator fusionOn-site workEntry-level Full Time InternshipCHN - Minhang, China20h ago
-
AI GPU Arch Perf Optimization Intern CNY 38K-50KCUDA | Computer Systems | GPU Kernels | GPU Programming | Memory systemsOn-site workEntry-level Full Time InternshipCHN - Minhang, China20h ago
-
AI GPU Arch Perf Optimization Intern CNY 38K-50KAI Fundamentals | Attention | CUDA | Computer Systems | GEMMCollaborative team environment | Internship experience | On-site workEntry-level Full Time InternshipCHN - Minhang, China20h ago
-
ATE | ATPG | Applied Machine Learning | C++ | DFTCross-functional collaboration | MentorshipSenior-level Full TimeChina, Shanghai20h ago
-
Software Engineering & Development, SrAssc CNY 38K-50KAWS | Azure | Data Engineering | Data analytics | Deep learningEmployee networks | Flexible work/life support | Inclusive development opportunities | Mentorship | Paid volunteer daysEntry-level Full TimeHangzhou, China20h ago
-
Entry-level Full Time北京 R23h ago
-
Miclaw-端云协同调度专家 (Hybrid AI Architect) CNY 240K-480K5G | Cloud Computing | Consistency protocols | Data Compression | Distributed SystemsHybrid work modelSenior-level Full Time北京 R1d ago
-
Mid-level Full Time武汉1d ago
-
Entry-level Full Time北京 R1d ago
-
Senior-level Full Time北京1d ago
-
Entry-level Full Time北京 R1d ago
-
Mid-level Full Time北京 R1d ago
-
Mid-level Full Time北京 R1d ago
-
ANSYS | APDL | C Programming | Design of Experiments | DynamicsNone Full TimeWuhan, Hubei, China1d ago
-
Senior-level Full TimeChina, Shanghai1d ago
-
IT Dept. AI Engineer_Application (上海) CNY 240K-360KAI machine learning | Alibaba Cloud | Cloud Applications | Database Design | Language ModelsMid-level Full TimeAnting, CN, 2018051d ago
-
Sr Machine Learning Engineer III CNY 240K-480KAPI Design | AWS | Agent Frameworks | Azure DevOps | CI/CDAdoption leave | Annual Medical Checkup | Family leave | Flexible benefits | Life insuranceSenior-level Full TimeChina-Shanghai (Tianshan-W-Rd)1d ago
-
Sr Machine Learning Engineer III CNY 240K-480KAPI Design | AWS | Agile | Azure DevOps | CI/CDAnnual Medical Checkup | Flexible benefits | Long service award | Medical and life insurance | Paid time offSenior-level Full TimeChina-Shanghai (Tianshan-W-Rd)1d ago
-
Recommendation Algorithm Engineer CNY 25K-37KDeep learning | Java | Recommendation Systems | SpringTeam collaborationEntry-level InternshipGuangzhou2d ago
-
Mid-level Full Time深圳2d ago
-
Mid-level Full Time上海2d ago
-
大数据开发（数据挖掘、数据测试、java） CNY 25K-37KApache Kafka | Apache Spark | Data Mining | Data Modeling | Data PipelinesMid-level Full Time保定2d ago
-
Entry-level Full Time广州2d ago