数据管线高级工程师
Tasks
- Build data lake ingestion workflow
- Build distributed data processing systems
- Build offline data processing
- Build real-time data processing
- Create data visualizations
- Design and build data pipeline core workflow
- Design data models
- Design unified data access and collaboration
- Develop data cleaning labeling quality check tools
- Develop data mining tools
- Expose data services
- Implement data cleaning and standardization
- Implement data collection and synchronization
- Implement data lineage tracking
- Implement data version control
- Implement metadata management and search
- Optimize data ingestion and transformation performance
- Optimize distributed system performance troubleshooting
- Provide production data support
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Apache Kafka | Caching | Data Lake | Data Mining | Data Modeling | Data Processing | Data Quality | Data Services | Data Warehousing | Data cleaning | Data labeling | Distributed Data Lake | Distributed Systems | Distributed data | Docker | ETL | Go | Java | Kubernetes | Lance | Metadata Management | MongoDB | MySQL | Offline data | Offline data processing | PostgreSQL | Pulsar | Python | Query engine | RabbitMQ | Real Time | Real-time Data | Real-time Data Processing | Redis | Stream processing
Education
Roles
Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Senior Machine Learning Engineer USD 20KData Analysis | Data Visualization | Deep learning | Experimentation | Fraud DetectionCareer growth | Competitive compensation | Open, collaborative cultureSenior-level Full TimeShenzhen, China11h ago
-
Senior Data Engineer / AI Engineer CNY 360K-600KAI Agents | Apache Spark | Azure | Data Modeling | DatabricksSenior-level Full TimeShanghai, Shanghai, China13h ago
-
Operations / IE Special II HKD 360K-550KAPI | C# | Data Loss Prevention | Data Modeling | Data VisualizationMid-level Full TimeHK - Hong Kong (HKCHU), Hong …14h ago
-
ARM | Android | C++ | Communication Protocols | Digital CommunicationSenior-level Full TimeShanghai, China1d ago
-
Apache Airflow | Apache Hadoop | Apache Kafka | Apache NiFi | BigQueryMid-level Full TimeGuangzhou, Guangdong, China1d ago
-
AVP - Data and Analytics CNY 300K-420KA/B | A/B Testing | Automation | B testing | Data CleansingFlexible working | Professional developmentExecutive-level Full TimeGuangzhou, Guangdong, China1d ago
-
AVP - Data and Analytics CNY 300K-420KA/B | A/B Testing | Automation | B testing | CRMFlexible working | Inclusive work culture | Professional developmentExecutive-level Full TimeGuangzhou, Guangdong, China1d ago
-
Analyst, Data Science CNY 144K-240KAI Governance | Computer Vision | Data analytics | Deep learning | GitSenior-level Full TimeCN-Dong Bao Building, China1d ago
-
Analyst, Data Science CNY 144K-240KAgile | Artificial Intelligence | Data Science | Deep learning | MLOpsSenior-level Full TimeCN-Dong Bao Building, China1d ago
-
Mid-level Full TimeShanghai, Shanghai, China1d ago
-
Mobile Power optimization Engineer CNY 108K-184KAnalog circuit | Analog circuit design | Android | BW Governor | C#Entry-level Full TimeShanghai, Shanghai, China1d ago
-
AI/ML | Agentic Engineering | Apache Airflow | Apache Kafka | Apache NiFiMid-level Full TimeGuangzhou, GD, CN, 5106201d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Remote work | Supportive team environmentMid-level Full TimeShenzhen, China2d ago
-
AI Agent算法架构师 CNY 144K-240KAgent Framework | Cloud Computing | Computer Vision | Edge AI | LLM InferenceSenior-level Full Time深圳3d ago
-
Entry-level Full Time北京3d ago
-
Mid-level Full Time上海3d ago
-
Apache Arrow | Data Engineering | Data Mining | Data Processing | Financial DataFast-paced environment | TeamworkMid-level Full TimeShanghai4d ago
-
Mid-level Full TimeBeijing, China4d ago
-
顶尖应届-大模型训练框架研发工程师-MiMo AI 基建 CNY 72K-96KC++ | CUDA | Data loading | DeepSpeed | Distributed ComputingEntry-level Full Time北京4d ago
-
Entry-level Full Time北京4d ago
-
Entry-level Full Time北京4d ago
-
Entry-level Full Time上海4d ago
-
Entry-level Full Time上海4d ago
-
顶尖应届-机器人具身智能算法研究员-机器人事业部 CNY 25K-37KC++ | Diffusion Models | Git | Imitation Learning | Knowledge DistillationEntry-level Full Time北京4d ago
-
顶尖应届-具身大模型数据工程师-机器人事业部 CNY 25K-37K3D Vision | Action Recognition | Apache Spark | CLIP | Cloud processingEntry-level Full Time北京4d ago