分布式计算与存储软件工程师
Tasks
- Build dataset management systems
- Collaborate with algorithm teams to deliver solutions
- Design multi source multimodal data ingestion
- Develop data loading tools
- Ensure data consistency and system stability
- Implement unified data loading transformations caching prefetching
- Improve performance for large scale data loading
- Manage relational and NoSQL metadata and caching
- Optimize columnar storage partition compression and vectorized reading
- Optimize data pipeline parallel reading multiprocessing and threading
- Perform distributed data processing performance tuning and troubleshooting
- Support custom sampling data augmentation and preprocessing
- Tune Linux file system and network I O for NFS and object storage
Perks/Benefits
- N/A
Skills/Tech-stack
Airflow | Apache Ray | C++ | Flink | Golang | Hadoop | Hive | Hugging Face | Hugging Face Datasets | Java | Kubeflow | Kubernetes | Linux | MongoDB | MySQL | NFS | NVIDIA CUDA | NVIDIA DALI | ORC | Object storage | Parquet | Petastorm | PostgreSQL | Prometheus | PyTorch Dataloader | Python | Redis | Spark | TensorFlow Datasets
Education
Related jobs
-
Ai算法实习生(振动与力学方向) CNY 25K-37KAPI Integration | Convolutional Neural Network | Keras | Langchain | Neural NetworkEntry-level Internship深圳7h ago
-
Associate Director, Data and Analytics Specialist CNY 240K-360KAgile | Ansible | Apache Spark | Bamboo | BitbucketMid-level Full TimeXi'an, Shaanxi, China16h ago
-
Advanced Software Engineer-C++ CNY 180K-300KC# | C++ | Concurrency | I/O | Interprocess CommunicationMid-level Full TimeShenyang - PIC, China1d ago
-
Advanced Software Engineer-C++ CNY 180K-300KC# | C++ | Concurrency | Data Flow | Data flow architectureMid-level Full TimeShenyang - PIC, China1d ago
-
Analyst, Data Science CNY 144K-240KApplication Integration | Debugging | Documentation | Java | JavaScriptSenior-level Full TimeCN-M Plaza, China1d ago
-
Senior Perception Engineer CNY 360K-600KAlgorithm Optimization | C++ | Computer Vision | Embedded Systems | Multi SensorDevelopment opportunities | Supportive work environmentSenior-level Full Time5-8F TOWER C, 788 JINZHONG ROAD, …1d ago
-
Principal Perception Engineer CNY 360K-600KC++ | Knowledge Distillation | Model Conversion | Model Pruning | Network CompressionSenior-level Full Time5-8F TOWER C, 788 JINZHONG ROAD, …1d ago
-
Senior Specialist, AI Application CNY 360K-600KAgile | Angular | Cloud Platforms | Generative AI | JavaSenior-level Full TimeCN-OCG International Center, Cheng Du, China1d ago
-
Senior-level Full TimeCN-OCG International Center, Cheng Du, China1d ago
-
Principal Specialist, AI Application CNY 240K-480KAgentic Workflows | Async Programming | Authentication | Authorization | Distributed SystemsSenior-level Full TimeCN-M Plaza, China1d ago
-
Principal Specialist, AI Application CNY 240K-480KAgentic Workflows | Asynchronous programming | Cloud Computing | Distributed Systems | DockerSenior-level Full TimeCN-M Plaza, China1d ago
-
Behavior Cloning | C++ | Cloud processing | Control | DaggerEntry-level Internship北京、上海 R1d ago
-
Entry-level Internship上海1d ago
-
真机强化学习实习生 CNY 25K-37KActor-critic | Deep Q Networks | Embodied Foundation Model | Foundation Model | Isaac-GymEntry-level Internship上海1d ago
-
Entry-level Internship上海1d ago
-
Entry-level Internship上海1d ago
-
具身智能数据开发实习生 CNY 25K-37KAPI Development | Algorithms | Data Structures | Data Transformation | Data VisualizationEntry-level Internship上海1d ago
-
Entry-level Internship上海1d ago
-
Mid-level Full Time上海1d ago
-
Mid-level Full Time深圳、上海、北京、中国香港1d ago
-
Mid-level Full Time深圳、上海、北京、中国香港1d ago
-
Mid-level Full Time深圳、上海、北京、中国香港1d ago
-
Entry-level Full Time深圳、北京、上海1d ago
-
Entry-level Full Time深圳、上海1d ago
-
大语言模型后训练算法工程师 CNY 240K-480KDistributed Training | Docker | Evaluation metrics | Fine Tuning | KubernetesMid-level Full Time深圳、上海1d ago