数据管线高级工程师
Guangzhou
CNY 240K-480K (estimate) Senior-level Full Time Internship
Tasks
- Build data lake ingestion and training data management platform
- Build offline and real time data processing systems
- Collaborate with machine learning teams to deliver data for model iteration
- Design and build data end to end pipeline
- Design data models and metadata management
- Develop data cleaning labeling quality check and data mining toolchains
- Develop distributed data storage and query solutions
- Implement data collection synchronization cleaning and standardization
- Implement data version control and lineage tracking
- Optimize large scale data transmission memory and IO performance
- Provide production and research data support
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Batch Processing | Caching | Columnar Storage | Data Lake | Data Mining | Data Modeling | Data Quality | Data Standardization | Data cleaning | Data labeling | Distributed Systems | Docker | ETL | GitHub | Go | Java | Kafka | Kubernetes | Lance | Log Collection | Metadata Management | MongoDB | MySQL | PostgreSQL | Pulsar | Python | Query engines | RabbitMQ | Real Time | Real-time Processing | Redis | Snapshot Isolation | Stream processing | Table Partitioning | Time processing
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Related jobs
-
(Sr) Cloud & Data Engineer CNY 192K-240KAWS | Automation | CI/CD | Container Security | Data ModelingMid-level Full TimeBeijing, Beijing, CN3h ago
-
大数据开发（数据挖掘、数据测试、java） CNY 25K-37KApache Kafka | Apache Spark | Data Mining | Data Modeling | Data WarehousingMid-level Full Time保定14h ago
-
Entry-level Full Time广州14h ago
-
Senior-level Full Time上海、北京15h ago
-
None Full Time淄博15h ago
-
None Full Time济南15h ago
-
【27届实习】Ai实习生(可转正) CNY 36K-48KAmazon Web Services | Computer Vision | Deep learning | Docker | KubernetesEntry-level Internship淄博、济南、青岛15h ago
-
Mid-level Full Time北京 R16h ago
-
Miclaw-端云协同调度专家 (Hybrid AI Architect) CNY 240K-360K5G | API Integration | Claude 3.5 | Distributed Systems | GPT-4oHybrid workSenior-level Full Time北京 R16h ago
-
Java开发工程师（大数据方向） CNY 180K-360KApache Flink | Apache Spark | Data pipeline | Distributed Systems | IO ProgrammingMid-level Full Time武汉17h ago
-
A/B | A/B Experimentation | Autoscaling | Caching | Canary testingCommute subsidy | Disability insurance | Employee stock ownership | Generous vacation | Health insuranceSenior-level Full TimeShanghai, China1d ago
-
Apache Airflow | Apache Flink | Apache Spark | Automated testing | Data LakeCommute subsidy | Competitive retirement pension plans | Employee resource groups | Employee stock ownership | Generous vacation personal daysSenior-level Full TimeShanghai, China1d ago
-
Airflow | CUDA | Data Lake | Data Warehouse | FlinkCommute subsidy | Competitive retirement pension plans | Employee resource groups | Employee stock ownership | Generous vacation personal daysSenior-level Full TimeShanghai, China1d ago
-
A/B | A/B Testing | Autoscaling | B testing | Canary testingCommute subsidy | Competitive retirement pension plans | Employee resource groups | Employee stock ownership | Generous vacationSenior-level Full TimeShanghai, China1d ago
-
Senior Data Engineer, Content Management Systems (China) CNY 144K-240KAPI Integration | AWS | Access Control | Alibaba Cloud | CI/CDAnnual medical check-up | Flexible benefits | Long service award | Medical and life insurance | Paid time offSenior-level Full TimeChina - Shanghai1d ago
-
AWS | Apache Airflow | Apache Kafka | Apache Spark | AzureMid-level Full TimeCN-Shenzhen-HyQ, China1d ago
-
Entry-level Internship Part TimeShanghai (JingAn), China1d ago
-
Mid-level Full Time深圳1d ago
-
Entry-level Internship北京1d ago
-
Senior Software Engineer - Machine Learning CNY 360K-600KData Analysis | Data Visualization | Deep learning | Experimentation | Fraud DetectionCareer progression | Collaborative culture | Competitive compensation | Global growth opportunitiesSenior-level Full TimeShenzhen, China1d ago
-
Senior Data Engineer CNY 156K-240KAccess Control | Agile | Apache Cassandra | Apache Hadoop | Apache KafkaSenior-level Full TimeBeijing, Beijing, China2d ago
-
AI intern CNY 28K-50KAutomated testing | Continuous integration | Deep learning | Generative AI | JavaEntry-level InternshipBeijing,Beijing,China2d ago
-
Intelligent Test Automation & GenAI Tool Engineer CNY 360K-540KAgent systems | C# | C++ | CI/CD | ConfluenceSenior-level Full TimeShanghai, Shanghai, China2d ago
-
Senior Data Engineer CNY 360K-600KActive Directory | Agile | Apache Spark | Azure Active Directory | Azure CosmosHybrid work environment | Inclusion support | Professional growth | Wellbeing supportSenior-level Full TimeChengdu, Manulife Information and Technology Center, …2d ago
-
Sr. AI Process Engineer, Seller Compliance CNY 360K-600KAWS | CI/CD | Data Pipelines | Deployment | Feature StoreSenior-level Full TimeShanghai, CHN3d ago