Big Data Engineer
Pittsburgh, Pennsylvania, United States
USD 100K-130K Senior-level Full Time
Tasks
- Build and optimize data workflows with Python and Spark
- Champion coding standards version control testing and documentation
- Collaborate with analysts scientists and stakeholders on technical designs
- Design data pipelines for ingestion transformation and processing
- Ensure data quality, integrity, and performance
- Identify opportunities to apply AI ML concepts to data engineering
- Manage and query data using Hadoop ecosystem components
- Mentor data engineers through code reviews and technical coaching
- Orchestrate batch scheduling with CA7 or Control-M
- Own data solution architecture and design decisions
- Troubleshoot and resolve production issues in data pipelines
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Spark | Batch scheduling | CA7 | Control-M | Data Processing | Data Quality | Distributed Systems | Documentation | ETL | HDFS | Hadoop | Hive | Impala | Job orchestration | Python | SQL | Testing | Version control
Education
N/A
Roles
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R29d ago
-
Business Intelligence | DBT | Data Integrity | Data Marts | Data SecurityPaid time offSenior-level Full TimeUnited States of America-OHIO-Franklin County-Columbus3h ago
-
Data/Business Analyst USD 115K-145KAI orchestration | API Gateway | AWS AgentCore | AWS Lambda | Amazon APIMid-level Full TimeUS-NY-New York3h ago
-
Data/Business Analyst USD 115K-145KAI Governance | AI orchestration | API Gateway | AWS AgentCore | AWS IAMMid-level Full TimeUS-NY-New York3h ago
-
Lead Principal AI Software Engineer USD 169K-355KAccess Control | AgentOps | Audit Logging | Autogen | Batching401k matching | Disability insurance | Employee stock purchase plan | Flexible spending accounts | Life insuranceSenior-level Full TimeNashville, TN, United States3h ago
-
Data Engineer II USD 92K-176KAWS | Airflow | Azure | Azure DevOps | Azure Synapse401k matching | Dental insurance | Employee resource groups | Health insurance | Onsite fitness centerSenior-level Full TimeUS-Rhode Island-Providence3h ago
-
Large Language Model Training System Engineer Graduate (Applied Machine Learning) - 2027 Start USD 128K-256KData Engineering | Distributed Training | Fine Tuning | Language Models | Large Language ModelsEntry-level Full TimeSan Jose, California, United States3h ago
-
Large Language Model Inference System Engineer Graduate (Applied Machine Learning) - 2027 Start USD 128K-256KCost Optimization | Distributed Systems | Elastic computing | Heterogeneous computing | KV cacheEntry-level Full TimeSan Jose, California, United States3h ago
-
Circuit breaking | Deep learning | Distributed Systems | Language Models | Language ProcessingEntry-level Full TimeSan Jose, California, United States3h ago
-
AIOps | APIs | Anomaly Detection | Automation | Cause analysisEntry-level Full TimeSan Jose, California, United States4h ago
-
Recommendation Architecture AI/ML Infrastructure Engineer Graduate (Data-Arch-TikTok Live) - 2027 Start USD 116K-177KAlgorithms | C++ | CUDA | Computer Systems | Continuous batchingEntry-level Full TimeSan Jose, California, United States4h ago
-
Software Engineer, Databases (Technical Leadership) USD 161K-261KAI Automation | Automated Performance Tuning | Consensus Protocols | Data Integrity | Database InternalsSenior-level Full TimeBellevue, WA | Menlo Park, CA5h ago
-
Machine Learning SoC Architect USD 173K-249KAI accelerators | ASIC Performance Modeling | C# | C++ | Computer ArchitectureSenior-level Full TimeSunnyvale, CA | Austin, TX | …5h ago
-
Staff Software Engineering, GenAI Applications, YouTube USD 207K-300KDeep learning | Dense retrieval | Embedding | Hybrid retrieval | Language ModelsSenior-level Full TimeSan Bruno, CA, USA5h ago
-
BIOS | C# | C++ | Co-development | Data StructuresHealth insurance | Paid time off | Professional development | Retirement plan | Stock optionsSenior-level Full TimeSunnyvale, CA, USA5h ago
-
Senior Software Engineer, AI/ML GenAI, Search USD 174K-252KC++ | Data Processing | Debugging | Distributed Systems | Information RetrievalSenior-level Full TimeMountain View, CA, USA5h ago
-
Research Engineer, Gemini Retrieval, DeepMind USD 174K-252KC plus plus | Information Retrieval | JAX | Language Models | Language ProcessingMid-level Full TimeMountain View, CA, USA; San Francisco, …5h ago
-
Senior, Data Analyst/BI Engineer USD 108K-145KAWS | Airflow | Amazon Redshift | BigQuery | DataStage401k matching | Healthcare dental vision coverage | PTO | Paid Company Holidays | Paid parental leaveSenior-level Full TimeUnited States R12h ago
-
Staff Machine Learning Engineer, Tools and Framework AI USD 175K-300KApplied Machine Learning | C++ | Java | Machine Learning | Machine Learning InfrastructureSenior-level Full TimeCupertino13h ago
-
Machine Learning Engineer USD 90K-210KAWS | Airflow | Azure | Computer Vision | Docker401k match | Dental insurance | Health insurance | Life insurance | Open leave policySenior-level Full TimeCambridge, MA, Arlington, VA, Seattle, WA13h ago
-
Bayesian Data Scientist USD 90K-210KAlgorithm Development | Bayesian statistics | Big Data | C# | C++10 paid holidays | 401k match | Dental insurance | Health insurance | Life insuranceMid-level Full TimeCambridge, MA, Arlington, VA14h ago
-
Applied AI and ML Lead - Generative AI USD 177K-215KAPI | Bias Testing | Continuous integration | Data Contracts | Data PipelinesBackup childcare | Financial coaching | Health care coverage | Mental health support | Retirement savings planSenior-level Full TimeJersey City, NJ, United States14h ago
-
Junior Data Engineer USD 70K-85KAWS | Amazon Redshift | Cloud BI | Data Cleansing | Data ModelingCasual work environment | Inclusive culture | Professional developmentEntry-level Full TimeRemote - Atlanta, GA, US R16h ago
-
Werkstudent als Data Engineer (m/w/d) USD 46K-48KAmazon Web Services | CI/CD | DBT | GitHub | JavaFlexible working hours | LinkedIn Learning account | Networking opportunities | Travel cost allowanceEntry-level Part TimeUnterföhring (bei München), DE, 8577416h ago
-
Distribution & Analytics GenAI Forward Deployed Engineer USD 145K-175KAPI Design | AWS | Amazon Bedrock | Amazon Bedrock AgentCore | Bedrock AgentCore401k | Dental insurance | Health insurance | Life insurance | Paid time offMid-level Full Time47006 Boston - 290 Congress St, …16h ago