Big Data Engineer
Tasks
- Build streaming data pipelines
- Design big data pipelines on Hadoop
- Design data models and storage layouts
- Develop ETL/ELT workflows
- Document data architectures and runbooks
- Evaluate and adopt new big data technologies
- Implement data governance lineage quality controls
- Lead performance reviews and architecture audits
- Mentor junior engineers
- Optimize Spark and MapReduce jobs
- Orchestrate pipeline workflows
- Set up monitoring alerting logging
Perks/Benefits
Skills/Tech-stack
AWS EMR | Airflow | Alerting | Apache Spark | Azure HDInsight | BigQuery | CI/CD | Data Governance | Data Lineage | Data Quality | Databricks | Delta Lake | Distributed Systems | Flink | HBase | HDFS | Hadoop | Hive | Hudi | Iceberg | Infrastructure as Code | Java | Kafka | Kubernetes | Logging | MapReduce | Monitoring | NoSQL | ORC | Oozie | Parquet | Pig | Python | SQL | Scala | Shell | Snowflake | Spark Streaming | Sqoop | Trino | “as-code”
Education
Roles
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R12d ago
-
Senior Data Engineer-JT0224 USD 115K-175K.Net Core | .Net Framework | AWS CloudFormation | Apache Airflow | Azure401k match | Career growth opportunities | Dental insurance | Employee resource groups | Health insuranceSenior-level Full TimeRemote, United States R9h ago
-
Senior Machine Learning Engineer USD 140K-224KAIOps | AWS | Apache Spark | Cloud platform | Data Analysis401k match | Dental insurance | Flexible paid time off | Life insurance | Medical insuranceSenior-level Full TimeRemote, United States R9h ago
-
Asynchronous training | Containerization | Curriculum learning | Distributed Systems | Distributed TrainingSenior-level Full TimeSF Bay Area, CA, Remote, US, … R13h ago
-
Machine Learning Engineer / AI Model Developer (Mid to Senior)(Top Secret Clearance Required) (Hybrid) USD 78K-176KAPIs | Agile methodologies | Backend Development | Cloud Computing | Data Preparation401k | Employee discount program | Employee referral rewards | Flexible spending account | Flexible work scheduleMid-level Full TimeFort Belvoir, VA, US R15h ago
-
Lead Software Engineer (Data)-7999 USD 129K-203KAmazon CloudWatch | Amazon EC2 | Amazon EMR | Amazon IAM | Amazon S3Comprehensive health benefits | Emotional wellbeing support | Financial support benefits | Personalized benefits | Remote work days per weekSenior-level Full TimeVA - Reston, 11951 Freedom Dr … R1d ago
-
Principal Advanced Analytics Analyst USD 123K-184KAgile | Data Modeling | Power Automate | Power BI | PowerApps401k matching | Dental insurance | Employee assistance program | Employee stock purchase plan | Flexible work scheduleSenior-level Full TimeUSA-MN, United States R1d ago
-
Sr. Software Engineer (Data)-0336 USD 132K-203KAWS | Apache Spark | ClickHouse | Databricks | GitHubEmotional support | Financial benefits | Health benefits | Remote work days per weekSenior-level Full TimeVA - Reston, 11951 Freedom Dr … R1d ago
-
Sr Staff Software Engineer- Data Platform USD 220K-255KAWS EMR | AWS Lambda | AWS S3 | Access Control | AlgorithmsDental insurance | Disability insurance | Flexible spending account | Health insurance | Health savings accountSenior-level Full TimeCalifornia - Remote Office, United States R1d ago
-
Senior Data Engineer USD 113K-254KAWS | Acceptance Testing | Azure | Best practices | CI/CD401k match | Career development opportunities | Charitable giving match | Dental insurance | Employee stock purchase planSenior-level Full TimeRemote, United States R1d ago
-
Lead Data Engineer USD 75KApache Airflow | Apache Beam | Autoscaling | BigQuery | CI/CD401k matching | Adoption reimbursement | Commuter benefits | Disability benefits | Discounts and savingsSenior-level Full Time112265-NJ-MetroPark, Iselin, United States R1d ago
-
Fullstack Engineer USD 85K-134KETL | Hadoop | Hive | Java | Linux401k plan | Continuing Education/Professional Development | Dental insurance | Employee assistance program | Life insuranceEntry-level Full TimeRemote, CA, United States R1d ago
-
Streaming Data Engineer USD 100K-150KAccess Control | Access Control Lists | Ansible | Apache Kafka | Apache Kafka ConnectSenior-level Full TimeUnited States - Remote R1d ago
-
Machine Learning Infrastructure Engineer USD 100K-150KAPI Gateway | Abuse detection | Automated rollback | Autoscaling | C++Senior-level Full TimeUnited States - Remote R1d ago
-
AI Scientist USD 100K-150KAccelerator hardware | Computer Vision | Deep learning | Distributed Training | Experiment designRemote workSenior-level Full TimeUnited States - Remote R1d ago
-
Machine Learning Data Engineer USD 100K-150KApache Beam | Apache Spark | CI/CD | Caching | Code reviewSenior-level Full TimeUnited States - Remote R1d ago
-
Generative AI Engineer USD 100K-150KAdapters | Benchmarking | DPO | Deep learning | Distributed TrainingSenior-level Full TimeUnited States - Remote R1d ago
-
Senior-level Full TimeUnited States - Remote R1d ago
-
LLM Engineer USD 100K-150KAdapter based Fine Tuning | Attention Optimization | Cluster management | DPO | Distributed TrainingSenior-level Full TimeUnited States - Remote R1d ago
-
Senior Director, Machine Learning & AI (BPD) USD 203K-304KAWS | Agentic Systems | Azure | Bias Testing | Cloud ComputingSenior-level Full TimeUS - Gaithersburg - MD, United … R1d ago
-
AI Optimization Engineer USD 100K-150KBenchmarking | C++ | CUDA | Continuous batching | CutlassRemote workSenior-level Full TimeUnited States - Remote R1d ago
-
Prompt Engineer USD 100K-150KAgentic Workflows | Chunking | Embeddings | Evaluation | IndexingMentorship opportunities | Remote workSenior-level Full TimeUnited States - Remote R1d ago
-
Senior-level Full TimeUnited States - Remote R1d ago
-
Robotics Engineer USD 100K-150KBehavior Trees | C++ | Cameras | Concurrent programming | Control SystemsSenior-level Full TimeUnited States - Remote R1d ago
-
Sr. Embedded Software/Firmware Engineer USD 180K-240KARM | ASPICE | AUTOSAR | C# | C++Competitive compensation | Dental insurance | Flexible hours | Medical insurance | Pre-IPO stock optionsSenior-level Full TimeUnited States - Remote R1d ago