分布式计算与存储软件工程师
Tasks
- Build dataset management systems
- Collaborate with algorithm teams to deliver solutions
- Design multi source multimodal data ingestion
- Develop data loading tools
- Ensure data consistency and system stability
- Implement unified data loading transformations caching prefetching
- Improve performance for large scale data loading
- Manage relational and NoSQL metadata and caching
- Optimize columnar storage partition compression and vectorized reading
- Optimize data pipeline parallel reading multiprocessing and threading
- Perform distributed data processing performance tuning and troubleshooting
- Support custom sampling data augmentation and preprocessing
- Tune Linux file system and network I O for NFS and object storage
Perks/Benefits
- N/A
Skills/Tech-stack
Airflow | Apache Ray | C++ | Flink | Golang | Hadoop | Hive | Hugging Face | Hugging Face Datasets | Java | Kubeflow | Kubernetes | Linux | MongoDB | MySQL | NFS | NVIDIA CUDA | NVIDIA DALI | ORC | Object storage | Parquet | Petastorm | PostgreSQL | Prometheus | PyTorch Dataloader | Python | Redis | Spark | TensorFlow Datasets
Education
Related jobs
-
Mid-level Full TimeChina - Hong Kong8h ago
-
Mid-level Full Time深圳、长沙17h ago
-
Mid-level Full Time深圳、上海、北京17h ago
-
具身数据 Infra Engineer CNY 144K-240KData Lineage | Data Quality | Data Validation | Data Versioning | Data pipelineEntry-level Full Time深圳17h ago
-
None Full Time深圳18h ago
-
视觉算法工程师(标定方向)-2027校招 CNY 180K-300KC++ | Camera Calibration | Camera Model | Computer Vision | Coordinate TransformationNone Full Time深圳18h ago
-
Mid-level Full TimeShanghai, Shanghai, China1d ago
-
Member of Technical Staff, Machine Learning CNY 144K-240KData Engineering | Data Pipelines | GPU Computing | JAX | Machine LearningSenior-level Full TimeChina1d ago
-
资深数据开发工程师(自动驾驶数据平台) _Xc CNY 180K-300KANN Index | COS Object Storage | Celery | ClickHouse | Cosine similarityMid-level Full TimeSuzhou, Jiangsu, China1d ago
-
Data Engineer - UCA CNY 144K-240KCloud Data | Cloud Data Platforms | Data Pipeline Monitoring | Data Warehousing | Data orchestrationSenior-level Full TimeShanghai1d ago
-
AI Compiler and Library Engineer CNY 45K-52KAssembly | C# | C++ | Computer Vision | Deep learningCollaborative environment | Hands-on project experience | On-site workEntry-level Full TimeCHN - Minhang, China1d ago
-
Senior Machine Learning Engineer CNY 457K-540KAPIs | Agentic AI | Bias Mitigation | Calibration | Class imbalanceSenior-level Full TimeShenzhen, Guangdong Province, China1d ago
-
Mid-level Full TimeShanghai, China1d ago
-
Mid-level Full TimeShanghai, China1d ago
-
ADAS | AI Agent | AI coding | AI coding assistant | Autonomous DrivingEmployee discounts | Employee events | Flexible working hours | Health measures | Hybrid workMid-level Full TimeShanghai, CN1d ago
-
ADAS Algorithm Engineer - Lateral Control 智驾横向控制算法工程师 CNY 360K-600KApollo | AutoWare | C++ | Code Management | Code reviewEmployee discounts | Employee events | Flexible working hours | Health measures | Hybrid workSenior-level Full TimeShanghai, CN1d ago
-
大数据开发工程师 CNY 180K-300KAI Agent | Anomaly Attribution | Apache Flink | Apache Spark | Data GovernanceMid-level Full Time北京1d ago
-
Omni模型算法工程师/研究员 CNY 500K-500KAgent planning | Audio Processing | Audio to Audio | C++ | Context understandingMid-level Full Time北京、深圳1d ago
-
AI院-GLM团队-AI-Native 全栈工程师(偏后端) CNY 180K-264KAPI Design | CSS | Cloud Native | Database operations | Distributed ComputingMid-level Full Time北京1d ago
-
Mid-level Full Time杭州1d ago
-
AI院--训练Infra工程师 CNY 180K-300KComputer Vision | Distributed Training | Language Models | Language Processing | Large Language ModelsEntry-level Full Time北京1d ago
-
Mid-level Full Time北京1d ago
-
MaaS-SRE/DBA CNY 240K-360KCapacity Planning | Cluster | Docker | Go | GrafanaOn-call duty | Performance optimization goalsSenior-level Full Time北京1d ago
-
Mid-level Full Time北京1d ago
-
Ai算法工程师实习 CNY 25K-37KComputer Vision | Deep learning | Hugging Face | Language Processing | Machine LearningFull-time internship | Long term internship preferred | One to one mentorshipEntry-level Internship合肥、上海1d ago