数据管线高级工程师
Tasks
- Build data lake ingestion workflow
- Build distributed data processing systems
- Build offline data processing
- Build real-time data processing
- Create data visualizations
- Design and build data pipeline core workflow
- Design data models
- Design unified data access and collaboration
- Develop data cleaning labeling quality check tools
- Develop data mining tools
- Expose data services
- Implement data cleaning and standardization
- Implement data collection and synchronization
- Implement data lineage tracking
- Implement data version control
- Implement metadata management and search
- Optimize data ingestion and transformation performance
- Optimize distributed system performance troubleshooting
- Provide production data support
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Apache Kafka | Caching | Data Lake | Data Mining | Data Modeling | Data Processing | Data Quality | Data Services | Data Warehousing | Data cleaning | Data labeling | Distributed Data Lake | Distributed Systems | Distributed data | Docker | ETL | Go | Java | Kubernetes | Lance | Metadata Management | MongoDB | MySQL | Offline data | Offline data processing | PostgreSQL | Pulsar | Python | Query engine | RabbitMQ | Real Time | Real-time Data | Real-time Data Processing | Redis | Stream processing
Education
Roles
Data Engineer | Data Engineering | Data Engineering Lead | Engineer | Engineering Lead | Lead
Related jobs
-
Lead AI Data Engineer CNY 144K-240KAmazon S3 | Apache Iceberg | Apache Spark | Azure Data | Azure Data LakeAnnual leave | Continuous learning culture | Digital wellbeing platform | Employee assistance program | Flexible working optionsSenior-level Full TimeGuangzhou, CN11h ago
-
AI平台与MLOps工程师(27届校招)_BCSC CNY 70KAccess Control | Alerting | Alibaba Cloud | Amazon Web Services | Argo CDEntry-level Full TimeWuxi, Jiangsu, China14h ago
-
Robotics & Automation Engineer CNY 360K-600K3D Software | AutoCAD | C++ | CAD | Lean ManufacturingRelocation assistanceSenior-level Full TimeSuZhou, China16h ago
-
Lead Robotics & Automation Engineer CNY 360K-600KAutoCAD | C++ | Lean Manufacturing | PLC programming | PTC CreoRelocation assistanceSenior-level Full TimeSuZhou, China16h ago
-
Data Platform Engineer CNY 277K-370KAWS | Apache Airflow | CI/CD | Data Modeling | Data PrivacyEmployee assistance programme | Flexible working | Learning and development | Life insurance | Medical insuranceSenior-level Full TimeGuangzhou, CN2d ago
-
AI Data Engineer- CNY 300K-420KAmazon S3 | Apache Spark | Azure Data | Azure Data Lake | Azure Data Lake StorageAnnual leave | Continuous learning culture | Digital wellbeing platform | Employee assistance programme | Flexible benefitsMid-level Full TimeGuangzhou, CN2d ago
-
Amazon CloudWatch | Amazon EKS | Ansible | Apache Airflow | Apache SparkSenior-level Full TimeShanghai2d ago
-
Apache Doris | Automation | Backup | CSI | Cloud ComputingSenior-level Full TimeShanghai2d ago
-
Apache Doris | Backup and Restore | CSI | CockroachDB | CouchbaseSenior-level Full TimeShanghai2d ago
-
Apache Flink | Apache Kafka | Apache Spark | CI/CD | JavaSenior-level Full TimeShanghai2d ago
-
Apache Doris | Backup and Restore | CSI | Cloud automation | CockroachDBSenior-level Full TimeShanghai2d ago
-
Apache Doris | Automation | Backup and Restore | CSI | CockroachDBSenior-level Full TimeShanghai2d ago
-
AWS | Alibaba Cloud | Automation | Backup and Restore | CSISenior-level Full TimeShanghai2d ago
-
Backup and Restore | CSI | CockroachDB | Disaster Recovery | GoSenior-level Full TimeShanghai2d ago
-
Bash | Cloud platform | Data Processing | Docker | Google CloudAsynchronous culture | Competitive compensation | Friendly work environment | Remote-friendlyMid-level Full TimeShanghai, China2d ago
-
资深机器视觉专家 CNY 180K-300KComputer Vision | Deep learning | Docker | Image Segmentation | Image classificationMid-level Full Time北京、上海2d ago
-
Entry-level Full Time北京2d ago
-
Senior-level Full Time常州2d ago
-
Hr智能化（共享服务） CNY 192K-300KAgent Framework | JavaScript | Language Model | Large Language Model | Low CodeMid-level Full Time北京2d ago
-
Mid-level Full Time北京2d ago
-
AI 平台高级 SRE 工程师 / DevOps 工程师 CNY 144K-240KAutomation | CUDA | Capacity Management | Containerd | Device pluginSenior-level Full Time北京2d ago
-
Senior-level Full Time北京2d ago
-
Mid-level Full Time北京2d ago
-
大模型规划算法工程师- 北京 CNY 240K-480KAgent architecture | Agent systems | Context Management | Deep learning | Dialogue UnderstandingMid-level Full Time北京2d ago
-
Senior-level Full Time InternshipGuangzhou2d ago