Apache Spark Developer
Tasks
- Build batch and real time ETL ELT pipelines
- Deploy Spark workloads on Databricks EMR Azure Synapse or Kubernetes
- Design distributed data processing applications using Apache Spark
- Develop reusable Spark libraries and metadata driven ingestion pipelines
- Implement data quality validation reconciliation monitoring and automated error handling
- Integrate Spark applications with data warehouses lakehouses and reporting platforms
- Modernize legacy ETL workloads into Spark based architectures
- Optimize Spark jobs for memory partitioning shuffle and execution efficiency
- Process structured semi structured and streaming data from databases APIs Kafka and cloud storage
- Transform and analyze data using PySpark Scala and Spark SQL
- Troubleshoot production distributed processing and cluster performance issues
Perks/Benefits
- N/A
Skills/Tech-stack
AWS Glue | Agile | Amazon EMR | Apache Hudi | Apache Iceberg | Apache Kafka | Apache Spark | Azure DevOps | Azure Event | Azure Event Hubs | Azure Synapse | Azure Synapse Analytics | Broadcast joins | CI/CD | Caching | Catalyst Optimizer | Cloud Dataproc | Databricks | Datadog | Dataframes | Datasets | Delta Lake | Docker | Event Hubs | Fault Tolerance | Git | GitHub Actions | Google Cloud | Google Cloud Dataproc | Grafana | HDFS | Hadoop | Hive | Jenkins | Kubernetes | OpenTelemetry | Oracle | Parquet | Partitioning | PostgreSQL | Prometheus | PySpark | RDD | SQL Server | Scala | Scrum | Shuffling | Snowflake | Spark SQL | Spark Structured Streaming | Structured Streaming | Synapse Analytics | Teradata | Terraform | Tungsten | YARN
Education
N/A
Roles
Apache Spark Developer | Data Engineer | Developer | Engineer | Spark Developer
Related jobs
-
Data Engineer CAD 100K-170KAWS | Apache Airflow | Azure | BigQuery | Cloud Computing401k | Asynchronous work culture | Autonomy | Company paid life insurance | Disability coverageMid-level Full TimeRemote - US or Canada R5h ago
-
Data Modeling & AI Analytics USD 133K-190KAirbyte | Amazon Redshift | BigQuery | DBT | Data GovernanceSenior-level Full TimeMiami, Florida R7h ago
-
Senior Research Engineer / Scientist - Storage for LLM USD 207K-368KCUDA | Caching | Data Structures | Distributed Systems | Eviction PolicyConference attendance | Open source contributions | Research resourcesSenior-level Full TimeSeattle, Washington, United States R8h ago
-
Research Engineer / Scientist - Storage for LLM USD 153K-300KCUDA | Caching | Distributed Systems | Eviction policies | GPU ComputingConference attendance | Innovation-driven culture | Open source contributions | Research resourcesEntry-level Full TimeSeattle, Washington, United States R8h ago
-
Data Engineer, Product Analytics USD 181K-204KAlgorithm Optimization | Bias Mitigation | C# | C++ | Data GovernanceCareer growth opportunitiesSenior-level Full TimeSunnyvale, CA | Bellevue, WA | … R9h ago
-
AI Security Engineer USD 140K-200K800-53 | ABAC | AI RMF | AI Studio | AKS401k matching | Dental insurance | Employee development program | Employee referral program | Medical insuranceSenior-level Full TimeRemote, MD, US R19h ago
-
Junior AI Engineer USD 86K-120KAPI Integration | Authentication | Cloud deployment | Docker | Error Handling401k matching | BYOD reimbursement | Company holidays | Flexible spending accounts | Health savings accountEntry-level Full TimeHouston, Texas, United States (Hybrid) R21h ago
-
Principal ML/AI Engineer USD 119K-274KAPI Development | Artificial Intelligence | Automated testing | Azure | CI/CDCareer growth | MentorshipSenior-level Full TimeREMOTE GA 3 - Remotely Working … R21h ago
-
AWS Bedrock | AWS Lambda | Amazon DynamoDB | Amazon S3 | Amazon SageMaker401k plan | Flexible time off | Global shutdown | Health insurance | Paid time offSenior-level Full TimePrinceton LVL - NJ, United States R21h ago
-
Principal Data Engineer (Remote) USD 151K-189KAmazon Kinesis | Apache Spark | CI/CD | Cloud Data | Cloud Data Services12 weeks new parent leave | Coworking support | Flexible PTO | Learning & development stipend | Time allotment for learning and developmentSenior-level Full TimeUnited States R1d ago
-
Lead Data Engineer (Remote) USD 126K-157KAWS | AWS Redshift | Amazon Kinesis | Azure | Azure DataCoworking support | Flexible PTO | Learning and development stipend | New parent leave | Remote workSenior-level Full TimeUnited States R1d ago
-
AWS | Agile | Azure | Azure Data | Azure Data FactoryHybrid work | Occasional onsite work | Onsite interviews as needed | Remote workSenior-level Full TimeUnited States R1d ago
-
AWS | Agile | Azure | Azure Data | Azure Data FactoryMid-level Full TimeUnited States R1d ago
-
Senior Analytics Engineer USD 141K-176KAWS Glue | Amazon Athena | Apache Hive | Apache Spark | CI/CDGenerous time off | Healthcare | Occasional travel for in person meetings | Paid personal time off | Paid sick timeSenior-level Full TimeRemote - US R1d ago
-
Senior Software Engineer (Forward Deployed) USD 180K-264KAWS | Automation | Data Contracts | Data Governance | Data QualityHybrid work | On-call support | Travel opportunitiesSenior-level Full TimeLondon - Hybrid; New York - … R1d ago
-
Data Scientist USD 90K-120KAWS | Agentic AI | Azure | Cypress | Data Analysis401k employer contribution | Remote work optionMid-level Full TimeRemote, USA R1d ago
-
Senior AI Engineer USD 170K-190KAPI Gateway | AWS Lambda | Amazon API | Amazon API Gateway | Amazon AuroraCareer advancement | Employee assistance program | Employee recognition | LinkedIn Learning Platform | Open PTOSenior-level Full TimeRemote US R1d ago
-
Senior Machine Learning Engineer I, AI & ML Platform USD 183K-205KAWS | Azure | DevOps | K8s | Kubernetes401k match | Child care backup care and elder care | Donation matching | Fertility care support | Fitness Mindfulness Nutrition and Sleep SubscriptionSenior-level Full TimeSan Francisco, CA (Hybrid) R1d ago
-
Data Analytics Engineer USD 130K-204KCommon table expressions | Data Modeling | Data Quality | Data Transformation | Data ValidationHealth insurance | Paid Company Holidays | Paid time offMid-level Full TimeUnited States R1d ago
-
Software Engineer, Storage USD 80K-105KAnsible | Ceph | Continuous Deployment | Continuous integration | Distributed Systems401k match | Anniversary bonus | Dental insurance premium paid by employer | Gym membership reimbursement | Health Insurance Premium Paid by EmployerMid-level Full TimeRemote - United States R1d ago
-
Senior Storage Engineer USD 130K-150KAutomation | Bash | Cause analysis | Ceph | Cloud infrastructure401k match | Company-paid health insurance | Gym membership reimbursement | Internet reimbursement | Paid sabbaticalSenior-level Full TimeRemote - United States R1d ago
-
Storage Operations Engineer USD 80K-100KAnsible | Automation | Bash | Block Storage | Ceph100 percent company paid insurance premiums | 401k match | Company paid Wellable subscription | Gym membership reimbursement | Internet reimbursementMid-level Full TimeRemote - United States R1d ago
-
Azure Data | Azure Data Factory | Azure DevOps | Azure Machine Learning | CI/CDRemote | USC OnlySenior-level Full TimeDallas, TX. R1d ago
-
ABAC | AWS CloudFormation | AWS Lambda | AWS Step Functions | AWS-APIsOnsite/hybrid workSenior-level Full TimeChantilly, Virginia R1d ago
-
Senior Site Reliability Engineer USD 168K-200KAWS | Alerting | Amazon S3 | Apache Kafka | AutoscalingSenior-level Full TimeRemote - United States R1d ago