大数据开发工程师
Tasks
- Build and maintain data assets catalog with metadata
- Create data service interfaces for BI and applications
- Define data warehouse architecture principles and development standards
- Define warehouse layers for write and query performance
- Design data warehouse dimensional models
- Develop batch data pipelines using Sqoop and ETL
- Develop data validation programs for data quality
- Develop real time stream processing jobs using Flink and Spark
- Integrate data sources including interfaces logs and message queues
- Monitor data output timeliness and optimize scheduling and SQL
- Optimize data models for query performance and lower maintenance
- Write ETL jobs using Hive and Spark SQL
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Flink | Apache Hive | Apache Spark | Data Asset | Data Asset Catalog | Data Modeling | Data Quality | Data Validation | Data Warehouse | Data Warehouse Architecture | Data Warehousing | DataWorks | Dimensional Modeling | Distributed Computing | Distributed Storage | DolphinScheduler | ETL | Java | Kettle | Metadata Management | Python | Real Time | Real-time stream processing | SQL | SQL Optimization | Shell | Spark SQL | Sqoop | Stream processing | Warehouse Architecture
Education
Roles
Big Data Developer | Data Developer | Data Engineer | Developer | Engineer
Related jobs
-
具身智能-强化学习(灵巧操作方向) 实习生 CNY 25K-37KActor-critic | Diffusion Models | Distributed Training | Embodied intelligence | Flow matchingEntry-level Full Time Internship深圳3h ago
-
DPO | Deep learning | Diverse Preference Optimization | Learning algorithms | Machine LearningMid-level Full Time上海5h ago
-
算法工程师-大模型数据方向 CNY 240K-360KAutomated Evaluation | Clustering | Corpus Synthesis | Data Augmentation | Data GovernanceSenior-level Full Time上海5h ago
-
数据开发工程师（Ai知识方向） CNY 180K-300KContent governance | Data Governance | Data Quality | Data Quality Metrics | ETLMid-level Full Time上海5h ago
-
Mid-level Full Time上海5h ago
-
Senior-level Full Time上海5h ago
-
Mid-level Internship上海5h ago
-
Mid-level Full Time上海5h ago
-
大语言模型后训练/Agentic算法工程师 CNY 180K-360KAgentic RL | DAPO | Distributed Training | Evaluation | Function CallingEntry-level Full Time上海、北京5h ago
-
Senior-level Full Time上海5h ago
-
Associate Director, Data and Analytics CNY 280K-360KApache Airflow | Automated testing | BigQuery | CI/CD | Cloud ComposerMid-level Full TimeGuangzhou, Guangdong, China13h ago
-
Entry-level Full TimeSuzhou, Jiangsu, China15h ago
-
AWS | Access Controls | Agile | Azure | CI/CDCareer growth opportunities | Continuous training | High-end technology access | Inclusive workplaceMid-level Full TimeCHN – Chengdu - Commercial, China23h ago
-
Senior System Software Engineer, Robotics CNY 144K-240KARM architecture | C# | C++ | CUDA | DeterminismSenior-level Full TimeChina, Shanghai23h ago
-
C plus plus | C# | Camera Calibration | Camera Synchronization | Camera systemsMid-level Full TimeShenzhen, Guangdong, China23h ago
-
Machine Learning Engineer CNY 216K-300KAndroid | C# | C++ | Embedded Systems | Inference OptimizationMid-level Full TimeShanghai, Shanghai, China23h ago
-
C plus plus | CUDA | Code generation | Compiler design | Domain-specific languageSenior-level Full TimeChina, Shanghai1d ago
-
Senior-level Full Time上海1d ago
-
Mid-level Full Time深圳1d ago
-
Ai算法工程师 CNY 180K-300KConvolutional Neural Networks | Data Mining | Data Warehouse | Data cleaning | Data labelingMid-level Full Time东莞1d ago
-
Ai 院--多模态团队--多模态理解算法研究员-强化学习方向 CNY 240K-480KDPO | Data Preprocessing | Data cleaning | DeepSpeed | Distributed TrainingSenior-level Full Time北京 R1d ago
-
AI院-GLM团队-AI-Native 全栈工程师（偏后端） CNY 180K-300KAPI Design | API design and implementation | Cloud Native | Data Processing | Database operationsMid-level Full Time北京1d ago
-
Mid-level Full Time杭州1d ago
-
AI院--训练Infra工程师 CNY 180K-300KComputer Vision | Distributed Training | Language Models | Language Processing | Large Language ModelsMid-level Full Time北京1d ago
-
Mid-level Full Time北京1d ago