分布式计算与存储软件工程师
Tasks
- Build dataset management systems
- Collaborate with algorithm teams to deliver solutions
- Design multi source multimodal data ingestion
- Develop data loading tools
- Ensure data consistency and system stability
- Implement unified data loading transformations caching prefetching
- Improve performance for large scale data loading
- Manage relational and NoSQL metadata and caching
- Optimize columnar storage partition compression and vectorized reading
- Optimize data pipeline parallel reading multiprocessing and threading
- Perform distributed data processing performance tuning and troubleshooting
- Support custom sampling data augmentation and preprocessing
- Tune Linux file system and network I O for NFS and object storage
Perks/Benefits
- N/A
Skills/Tech-stack
Airflow | Apache Ray | C++ | Flink | Golang | Hadoop | Hive | Hugging Face | Hugging Face Datasets | Java | Kubeflow | Kubernetes | Linux | MongoDB | MySQL | NFS | NVIDIA CUDA | NVIDIA DALI | ORC | Object storage | Parquet | Petastorm | PostgreSQL | Prometheus | PyTorch Dataloader | Python | Redis | Spark | TensorFlow Datasets
Education
Related jobs
-
Mid-level Full TimeShanghai5h ago
-
【校招实习生】系统研发工程师 CNY 75K-75KAI Agent | Alertmanager | ArgoCD | CI/CD | Cloud ComputingAI development tools access | Hands-on project experience | Internal training | One to one mentorship | Technical workshopsEntry-level Internship北京5h ago
-
Senior Software Engineer (RAG Backend Developer) CNY 277K-370KA/B | A/B Testing | ABAC | Access Control | Audit LoggingSenior-level Full TimeGuangzhou, Guangdong, China R19h ago
-
Asynchronous Communication | C plus plus | Continuous batching | Data parallelism | Deep learningOn-site workEntry-level Full TimeCHN - Minhang, China23h ago
-
Senior Perception Engineer CNY 144K-240KBEV perception | C++ | Computer Vision | Data pipeline | Deep learningSenior-level Full Time5-8F TOWER C, 788 JINZHONG ROAD, …23h ago
-
Mid-level Full Time北京1d ago
-
None Full Time上海1d ago
-
Entry-level Internship上海1d ago
-
AI software application developers (CG) CNY 84K-120KAI | Algorithms | Benchmarking | C# | C++Career growth | Continuous learning | Onsite workEntry-level Full TimeCHN - Minhang, China1d ago
-
AI Framework Software Engineer - vLLM CNY 300K-420KAsynchronous Communication | C++ | Data parallelism | Deep learning | Distributed SystemsEntry-level Full TimeCHN - Minhang, China1d ago
-
Mid-level Full TimeXian, Shaanxi, China1d ago
-
Mid-level Full TimeShanghai, China1d ago
-
AI Frameworks Engineer - Model and Kernel Optimization CNY 160K-240KC++ | Deep learning | GPU Programming | Kernel development | PyTorchOn-site workSenior-level Full TimeCHN - Minhang, China1d ago
-
AI Framework Software Engineer – PyTorch CNY 74K-100KC++ | Debugging | Deep learning | Distributed Computing | GPU OptimizationOn-site workEntry-level Full TimeCHN - Minhang, China1d ago
-
AI Compiler and Library Engineer CNY 60K-60KAssembly | C# | C++ | Deep learning | JavaCollaborative environment | Hands-on projects | On-site workEntry-level Full TimeCHN - Minhang, China1d ago
-
Data Analyst Engineer A1 CNY 360K-510KAgile | Bash | CI/CD | CSS | CloudEmployee networks | Flexible work/life support | Inclusive development opportunities | Paid volunteer daysSenior-level Full TimeHangzhou, China1d ago
-
Senior-level Full TimeCHN - ZJ - Hangzhou, Innovation …1d ago
-
Mid-level Full Time北京2d ago
-
AI基础设施研发工程师（Sandbox / 容器化)-MiMo CNY 144K-240KAlerting | AppArmor | Argo Workflows | CI/CD | CPU SchedulingMid-level Full Time北京2d ago
-
Senior-level Full Time北京2d ago
-
Algorithms | Artificial Intelligence | Concurrency | Containerization | Data StructuresMid-level Full TimeShanghai2d ago
-
Apache Cassandra | Apache Solr | CI/CD | Elasticsearch | JavaSenior-level Full TimeShanghai2d ago
-
Apache Flink | Apache Kafka | Apache Spark | Apache Superset | Data GovernanceMid-level Full TimeShanghai2d ago
-
Machine Learning Engineer CNY 180K-360KAndroid | C# | C++ | Convolutional Neural Network | Deep learningEntry-level Full TimeShanghai, China2d ago
-
.NET 6 | .Net 8 | .Net Framework | Asynchronous programming | C#Mid-level Full TimeChina-Shanghai2d ago