Senior Data Engineer - Hadoop/Spark/OCP/Big Data
Two PNC Plaza (PA374), United States
USD 50K-185K Senior-level Full Time
Tasks
- Automate deployments, testing, and monitoring
- Build ETL and ELT frameworks
- Contribute to data modeling and solution design
- Deploy and manage data workloads on Docker and Kubernetes
- Design and maintain scalable data pipelines
- Ensure data governance, security, and compliance
- Implement CI CD pipelines for data engineering workflows
- Maintain technical documentation and conduct knowledge transfer
- Manage data in object storage and data lakes
- Monitor cluster resources and resolve processing bottlenecks
- Optimize Spark jobs for performance and resource efficiency
- Optimize table formats for large scale analytics
- Perform code review and change management
- Provide technical leadership and mentor engineers
- Support data migration and modernization
Perks/Benefits
- N/A
Skills/Tech-stack
Apache Iceberg | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | CI/CD | CPU Memory Storage Monitoring | Data Lake | Data Lake Storage | Data Lifecycle Management | Data Modeling | Data Partitioning | Data Versioning | Data lifecycle | Distributed Computing | Docker | ELT | ETL | GitHub | GitHub Actions | Infrastructure as Code | Jenkins | Kubernetes | Lifecycle Management | OpenShift | Python | Resource Optimization | S3 | SQL | Spark | Storage Monitoring | Table formats | “as-code”
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
Acceptance Testing | Business Objects | Cause analysis | Data Visualization | Data WarehousingOn-site work | Security clearance supportSenior-level Full TimeReston, VA; Information Technology3h ago
-
Amazon Redshift | Data Modeling | Data Quality | Data Warehouse | Data Warehouse ArchitectureOnsite work at client site | Paid on time | Potential project extensions | Support from consultantsMid-level Full TimeJuno Beach, FL; Juno Beach, FL; …3h ago
-
Lead Software Engineer / Data Analyst - (MUST be eligible for a TS/SCI security clearance) - 1822 A USD 115K-185K.NET | AWS | Angular | Automated testing | CI/CDEvery other Friday off on 9 80 schedule | Onsite work | Relocation assistanceSenior-level Full TimePatrick Space Force Base, FL3h ago
-
API Design | BigQuery | CI/CD | Chunking | Cloud BuildHybrid work | Local preferred | Onsite 4 days per weekMid-level Full TimeDearborn, United States R5h ago
-
Agent Orchestration | Agentic AI | Airflow | Autogen | BentoMLSenior-level Full TimeCharlotte, United States6h ago
-
Agile | Azure Databricks | Azure Logic | Azure Logic Apps | Azure SynapseEntry-level Full Time InternshipPella, IA, United States6h ago
-
Acceptance Testing | Agile | Artificial Intelligence | Azure DevOps | ConfluenceSenior-level Full TimeRosslyn, Virginia, United States7h ago
-
PROJECT - Lead Analytics and Insights Engineer II USD 137K-229KAcceptance Testing | Agile | Artificial Intelligence | Azure DevOps | ConfluenceSenior-level Full TimeRosslyn, Virginia, United States7h ago
-
AI Foundry | Agent systems | Agentic AI | Amazon Web Services | Artificial IntelligenceMid-level Full TimeMechanicsburg, Pennsylvania, United States7h ago
-
Intelligent Automation Manager | TS/SCI USD 137K-229KAWS Bedrock | AWS Textract | Agile | Amazon Web Services | Data ArchitectureSenior-level Full TimeRosslyn, Virginia, United States7h ago
-
Senior/Staff LLM Application Engineer - Data Application USD 108K-190KAgentic Systems | Benchmarking | Data Querying | Data analytics | Fine TuningSenior-level Full TimeSan Jose, California, United States7h ago
-
Action Conditioned World Models | C plus plus | Computer Vision | Deep learning | Dexterous ManipulationOpen science contributionsEntry-level Full TimeMenlo Park, CA8h ago
-
AI workflows | Anomaly Detection | Data Architecture | Data Lakehouse | Data StorageSenior-level Full TimeMenlo Park, CA8h ago
-
Research Engineer, Frontier AI, Incubation, DeepMind USD 174K-252KAlgorithm Design | C++ | Compute Optimization | Data Structures | Distributed SystemsEntry-level Full TimeMountain View, CA, USA8h ago
-
C++ | Data Processing | Data Structures | Data Structures and Algorithms | DebuggingSenior-level Full TimeMountain View, CA, USA; Pittsburgh, PA, …8h ago
-
Artificial Intelligence | C++ | CSS | Data Storage | Distributed ComputingHealth insurance | Paid time off | Parental leave | Retirement plansSenior-level Full TimeSunnyvale, CA, USA8h ago
-
Senior Data Engineer, gTech Risk USD 156K-226KApache Flume | Apache Spark | Data Modeling | Data Pipelines | Data VisualizationSenior-level Full TimeChicago, IL, USA8h ago
-
Staff Software Engineer, Data Protection USD 207K-300KAccess Control | Big Data | Big data analytics | C# | C++Senior-level Full TimeNew York, NY, USA8h ago
-
Software Engineer, AI/ML, Computer Vision, XR USD 147K-210K3D Reconstruction | Android | Auto-exposure | C++ | Computer VisionMid-level Full TimeSan Jose, CA, USA8h ago
-
Senior Software Engineer, Data Engineering, Cloud AI USD 174K-252KAccess Control | Alerting | Artificial Intelligence | Batch Processing | Cloud StorageSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA8h ago
-
Data Engineer USD 220K-226KC++ | Data Exploration | Data Modeling | Data Visualization | Data pipelineHybrid scheduleMid-level Full TimeMountain View, CA, USA R8h ago
-
Software Engineer, Generative AI, Shopping on Gemini USD 147K-210KBackend Development | C++ | Computer Vision | Data Preparation | Data ProcessingMid-level Full TimePittsburgh, PA, USA; Mountain View, CA, …8h ago
-
AI Solution Security Engineer USD 147K-210KAccess Control | C++ | Cloud Security | Data Flow | Data Flow SecurityMid-level Full TimeChicago, IL, USA8h ago
-
Software Engineer USD 149K-175KAlgorithms | C# | C++ | Cause analysis | Code ReviewsBenefits | Hybrid scheduleEntry-level Full TimeMountain View, CA, USA R8h ago
-
C++ | Clustering | Computer Vision | Data Processing | Data StorageSenior-level Full TimePittsburgh, PA, USA8h ago