数据管线高级工程师
Tasks
- Build data lake ingestion workflow
- Build distributed data processing systems
- Build offline data processing
- Build real-time data processing
- Create data visualizations
- Design and build data pipeline core workflow
- Design data models
- Design unified data access and collaboration
- Develop data cleaning labeling quality check tools
- Develop data mining tools
- Expose data services
- Implement data cleaning and standardization
- Implement data collection and synchronization
- Implement data lineage tracking
- Implement data version control
- Implement metadata management and search
- Optimize data ingestion and transformation performance
- Optimize distributed system performance troubleshooting
- Provide production data support
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Apache Kafka | Caching | Data Lake | Data Mining | Data Modeling | Data Processing | Data Quality | Data Services | Data Warehousing | Data cleaning | Data labeling | Distributed Data Lake | Distributed Systems | Distributed data | Docker | ETL | Go | Java | Kubernetes | Lance | Metadata Management | MongoDB | MySQL | Offline data | Offline data processing | PostgreSQL | Pulsar | Python | Query engine | RabbitMQ | Real Time | Real-time Data | Real-time Data Processing | Redis | Stream processing
Education
Roles
Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Behavior Cloning | C++ | Cloud processing | Control | DaggerEntry-level Internship北京、上海 R10h ago
-
Entry-level Internship上海10h ago
-
Entry-level Internship上海10h ago
-
Entry-level Full Time深圳、北京、上海11h ago
-
Entry-level Full Time深圳、上海11h ago
-
大语言模型后训练算法工程师 CNY 240K-480KDistributed Training | Docker | Evaluation metrics | Fine Tuning | KubernetesMid-level Full Time深圳、上海11h ago
-
Mid-level Full Time上海13h ago
-
Mid-level Full Time北京 R14h ago
-
Miclaw-端云协同调度专家 (Hybrid AI Architect) CNY 240K-360KClassification | Cloud Computing | Consistency protocols | Data Compression | Distributed SystemsFull-time employment | Hybrid work environmentSenior-level Full Time北京 R14h ago
-
Entry-level Internship北京14h ago
-
【集团】数据库运维工程师 CNY 240K-360KAutomation tools | Backup and Recovery | Data synchronization | Database Migration | Database monitoringMid-level Full Time Temporary上海15h ago
-
Entry-level Full Time上海15h ago
-
Entry-level Full Time上海15h ago
-
Avp - Data And Analytics CNY 300K-420KAWS | BigQuery | Cloud Computing | Cloud platform | Customer SegmentationCareer growth opportunities | Flexible working | Inclusive work environment | Professional developmentExecutive-level Full TimeGuangzhou, Guangdong, China R22h ago
-
AI/ML Scientist CNY 300K-420KAPI Development | Computer Vision | Data Analysis | Data Modeling | Data PreprocessingEntry-level Full TimeCNSGH18 - Shanghai - No. 757 …1d ago
-
Data Analysis Engineer-AI CNY 192K-300KAmazon Web Services | Apache Spark | Big Data | Data Analysis | Data MiningSenior-level Full TimeCQM01 - CQM01, Chongqing Software Park, …1d ago
-
Data Analysis Engineer CNY 300K-420KAlgorithms | Amazon Web Services | Apache Spark | Big Data | Data MiningMid-level Full TimeCQM01 - CQM01, Chongqing Software Park, …1d ago
-
大模型算法实习生 CNY 36K-37KDeep learning | DeepSpeed | Distributed Training | Java | Machine LearningCollaborative team | Large NLP dataset access | Long term internship support | Technical mentorship | Technical resourcesEntry-level Internship北京、上海1d ago
-
Mid-level Full Time北京1d ago
-
Entry-level Full Time北京、广州、上海1d ago
-
Entry-level Internship深圳1d ago
-
Computer Vision | Computer Vision Analytics | Data Annotation | Data Engineering | Data PipelinesSenior-level Full TimeSuzhou, China2d ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeAsynchronous culture | Flexible managementMid-level Full TimeBeijing, China2d ago
-
Senior Embeded QA Engineer CNY 120K-180KAPI Testing | Agile | Alarms | Ant | BACnetFlexible time off | Paid parental leave | Vacation and holiday leaveSenior-level Full TimeXi'an, China2d ago
-
Deep Learning Compiler CI/Infrastructure Engineer CNY 160K-240KAI Agents | Agent workflows | Artifact management | Automated triage | AutomationGenerous benefits packageSenior-level Full TimeChina, Shanghai R2d ago