Senior Data Engineer - Hadoop/Spark/OCP/Big Data
Two PNC Plaza (PA374), United States
USD 50K-185K Senior-level Full Time
Tasks
- Automate deployments, testing, and monitoring
- Build ETL/ELT frameworks
- Conduct knowledge transfer sessions
- Contribute to data architecture and pipeline design
- Deploy and manage data workloads with containers
- Design and maintain scalable data pipelines
- Design and manage data lakes on object storage
- Ensure data governance, security, and compliance
- Implement CI CD pipelines for data engineering workflows
- Implement and optimize Apache Iceberg tables
- Maintain technical documentation
- Monitor and improve resource utilization
- Optimize spark job performance
- Provide technical leadership and mentoring
- Support data migration and modernization
- Use infrastructure as code
Perks/Benefits
Skills/Tech-stack
Apache Iceberg | Azure Data | Azure Data Lake | Azure Data Lake Storage | Azure DevOps | Batch Processing | Bitbucket | CI/CD | Code review | Data Governance | Data Lake | Data Lake Storage | Data Partitioning | Distributed Computing | Docker | ELT | ETL | GitHub | Infrastructure as Code | Jenkins | Kubernetes | Lifecycle Management | Object storage | OpenShift | Performance Tuning | Real Time | Real-time Processing | Resource Optimization | S3 | SQL | Security Compliance | Spark | Terraform | Time processing | “as-code”
Education
Roles
Regions
Countries
States
Cities
Related jobs
-
Featured Feat. Principal Knowledge & Data Architect USD 174K-284KAWS Neptune | Canonicalization | Chunking | Cypher | DBTBenefits including health and wellness programs | Health, wellness, and retirement plansSenior-level Full TimeHeadquarters - Chevy Chase, MD R6d ago
-
Data Engineer USD 110K-165KCause analysis | Data Analysis | Data Pipelines | Data Processing | Data TransformationSenior-level Full TimeFAIRFAX, VA, United States10h ago
-
Lead Data Engineer USD 114K-160KBusiness Objects | Business Process | Business Process Redesign | Case Development | Compliance AuditingSenior-level Full TimeColumbus, GA, United States11h ago
-
Azure Data -AI -Architect USD 119K-258KAzure Compliance | Azure Data | Azure Data Factory | Azure Databricks | Azure DevOpsSenior-level Full TimeUSA12h ago
-
AWS | Cloud Computing | Columnar Storage | Dataframe optimization | JavaEquity | Full benefits | Hybrid work in office component required | Performance-focused culture | Startup environmentSenior-level Full TimeLos Angeles, CA, United States12h ago
-
Senior-level Full TimeJersey City, NJ, United States14h ago
-
Applied AI/ML [Multiple Positions Available] USD 165K-195KAI Governance | AI governance framework | AWS | Azure | BM25Backup childcare | Financial coaching | Health care coverage | Mental health support | On-site health and wellness centersSenior-level Full TimeJersey City, NJ, United States14h ago
-
AWS | AWS EMR | AWS Glue | Amazon S3 | Apache AirflowSenior-level Full TimePrinceton14h ago
-
Alerting | Data Governance | Data Modeling | Data Quality | Data Versioning401k match | Dental insurance | Life insurance | Long-term disability | Medical insuranceSenior-level Full TimePrinceton15h ago
-
Principal Software Engineer USD 105K-159KAWS | Agile | Azure | Bitbucket | CI/CDFlexible hybrid work model | Health and life insurance | Paid time off | Pension/retirement benefits | Personal/Family Care leavesSenior-level Full TimeTampa, FL, United States R15h ago
-
Sr. Robotics Engineer USD 124K-186KAgile | Bash | Build Servers | C# | C++401k | Adoption Assistance | Charitable contribution match | Defined contribution plan | Discount programsSenior-level Full TimeSanta Clara,California,United States15h ago
-
Senior Software Engineer, Infrastructure, Google Cloud USD 174K-253KApache Beam | Apache Spark | C++ | Compute architecture | Data AnalysisSenior-level Full TimeSunnyvale, CA, USA16h ago
-
Cloud Technical Solutions Engineer, Storage and Database USD 150K-218KC# | C++ | Docker | HTTP | Java24x7 support coverageSenior-level Full TimeSunnyvale, CA, USA; Austin, TX, USA16h ago
-
Software Engineer III, Cloud Bigtable SQL and Analytics USD 147K-211KApproximate Nearest Neighbors | Beam | C++ | Concurrency | Consistency modelsSenior-level Full TimeNew York, NY, USA16h ago
-
Algorithms | C++ | Cloud Storage | Data Analysis | Data StructuresSenior-level Full TimeSunnyvale, CA, USA16h ago
-
Senior Staff Engineer, GDC AI Inference Platform USD 262K-365KAgent coordination | C++ | Cloud infrastructure | Containerization | Data StructuresBonus | Equity | Health insurance | Paid time off | Retirement plansSenior-level Full TimeSunnyvale, CA, USA; Kirkland, WA, USA16h ago
-
Business Intelligence - Data Engineer 133-2015 USD 80K-115KData Transformation | Data analytics | Data pipeline | Dependency management | ETLMid-level Full TimeTulsa, OK, US18h ago
-
Sr AI Engineer USD 115K-142KAgent systems | Agentic Workflows | CI/CD | Citation Grounding | Cloud ComputingSenior-level Full TimeAda, MI, United States, 4935520h ago
-
PIIM Analytics Engineer USD 86K-132KAPIs | Access Control | Azure Databricks | Data Architecture | Data GovernanceDental insurance | Medical insurance | Optional Disability Insurance | Paid time off | Supplemental insuranceMid-level Full TimeClay, NY, US22h ago
-
PIIM Analytics Engineer USD 105K-160KAI/ML | APIs | Access Control | Azure Databricks | Code AutomationFull time office or project telework | Relocation assistanceMid-level Full TimeHouston, TX, US22h ago
-
Senior Robotics Systems Engineer USD 121K-165KActuator integration | Cause analysis | Circuit design | Debugging | Device Drivers401k plan | Dental insurance | Fully stocked kitchen | Green card support | Health insuranceSenior-level Full TimeMilpitas, CA23h ago
-
Principal AI Engineering Architect USD 224K-308KAWS | Data Processing | Docker | Google Cloud | High AvailabilityAdoption and surrogacy reimbursement | Cancer Care Program | Employee Assistance Program (EAP) | Fertility coverage | Flexible time offSenior-level Full TimeUnited States - Remote R23h ago
-
Customer Support Engineer USD 140K-270KBash | File systems | High Performance | High-Performance Computing | LinuxHoliday coverage | Rotational weekend shift | Travel opportunitiesSenior-level Full TimeDallas, Texas23h ago
-
Customer Support Engineer USD 140K-270KBash | File systems | High Performance | High-Performance Computing | LinuxOccasional holiday coverage | Rotational weekend shift | Travel opportunitiesSenior-level Full TimeHouston, Texas23h ago
-
Customer Support Engineer USD 140K-270KBash | File systems | High Performance | High-Performance Computing | LinuxHoliday coverage | Rotational weekend shift | Travel opportunitiesSenior-level Full TimeSan Francisco, CA, US23h ago