Apache Spark Developer
Tasks
- Build batch and real time ETL ELT pipelines
- Deploy Spark workloads on Databricks EMR Azure Synapse or Kubernetes
- Design distributed data processing applications with Apache Spark
- Develop data transformations using PySpark Scala and Spark SQL
- Develop reusable Spark libraries and metadata driven ingestion pipelines
- Implement data quality validation reconciliation and monitoring
- Integrate Spark with data warehouses lakehouses and reporting platforms
- Modernize legacy ETL workloads into Spark based architectures
- Optimize Spark job performance and execution efficiency
- Participate in architecture reviews code reviews and Agile development
- Process structured semi structured and streaming data
- Troubleshoot production distributed processing and cluster performance issues
Perks/Benefits
- N/A
Skills/Tech-stack
ARM Templates | AWS EMR | AWS Glue | AWS Step Functions | Amazon S3 | Apache Airflow | Apache Hudi | Apache Iceberg | Apache Kafka | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure Databricks | Azure DevOps | Azure Synapse | Azure Synapse Analytics | Broadcast Join | CI/CD | Caching | Catalyst Optimizer | Cloud Dataproc | Cloud Storage | Control-M | Data Factory | Data Lake Storage Gen2 | Data Lakehouse | Data Warehousing | DataFrame | DataSet | Datadog | Deequ | Delta Lake | Dimensional Modeling | Docker | Event Hubs | Fault Tolerance | Feature Engineering | Git | GitHub Actions | Google Cloud | Google Cloud Dataproc | Google Cloud Storage | Grafana | Great Expectations | HDFS | Hadoop | Hadoop Ecosystem | Hive | Jenkins | Kubernetes | Lakehouse Architecture | MLflow | MLlib | OpenTelemetry | Oracle | Parquet | Partitioning | Performance Tuning | PostgreSQL | Presto | Prometheus | PySpark | RDD | SQL | SQL Performance | SQL Performance Tuning | SQL Server | Scala | Shuffling | Snowflake | Spark SQL | Spark Structured Streaming | Step Functions | Structured Streaming | Synapse Analytics | Teradata | Terraform | Trino | Tungsten engine | YARN
Education
Bachelor of Engineering | Bachelor of Science | Master of Science
Roles
Related jobs
-
Azure | DAX | Data Modeling | Dataflows | ETL401k matching | Counseling sessions | Dental insurance | Disability coverage | Employee assistance programMid-level Full TimeUS-PR-AGUADILLA-110 ~ Rd 110 N Km … R2d ago
-
Software Engineer, AI/ML USD 102K-160KAWS | AWS Bedrock | Azure | Azure OpenAI | C#401k matching | Disability insurance | Employee ownership | Growth opportunities | Health insuranceSenior-level Full TimeUSA-GA-Office-Atlanta, United States R3d ago
-
Lead AI Engineer - Remote USD 145K-249KAWS Bedrock | Agent Orchestration | Agentic AI | Docker | FastAPI401k contribution | Comprehensive benefits package | Remote work within the United StatesSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R3d ago
-
Senior Power BI Developer USD 91K-163KAccess Control | Active Directory | Azure Active Directory | Azure Data | Azure Data Lake401k contribution | Career development opportunities | Comprehensive benefits package | Equity stock purchase | Incentive and recognition programsSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R3d ago
-
Apache Spark Developer USD 125K-185KARM Templates | AWS Glue | AWS Step Functions | Agile Scrum | Amazon EMRSenior-level Full TimeUnited States - Remote R3d ago
-
ETL Developer USD 115K-175KAWS | AWS Glue | Apache Airflow | Auditability | AutosysCareer growth potential | Remote workSenior-level Full TimeUnited States - Remote R3d ago
-
Sr. Optimizely Developer USD 130K-175K.Net Core | A/B | A/B Testing | API Integration | Adobe CommerceSenior-level Full TimeRemote, MA, US R4d ago
-
AI Solutions Developer USD 113K-170KAPI Development | AWS | Artificial Intelligence | Backend Development | Cloud Computing401k matching | Adoption Assistance | FSA | Fertility assistance | HSAEntry-level Full TimeCHICAGO, United States R4d ago
-
AI Solutions Developer USD 113K-170KAPI Development | AWS | Backend Development | Cloud Computing | Data Governance401k employer match | Adoption and fertility assistance | Background check required | Choice time off | FSA and HSA accountsEntry-level Full TimeCHICAGO, United States R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Creative Analytics | Customer Acquisition | Customer Acquisition Cost | DashboardingCareer growth path | Education and best practice support | Remote-first culture | Self service analytics enablementSenior-level Full TimeAsheville, NC (remote); Atlanta, GA (remote); … R5d ago
-
Lead Data Analyst, Marketing USD 120K-150KAd-Hoc | Ad-hoc analysis | Affiliate Marketing | Dashboard Development | Data ModelingCareer growth path | Competitive benefits | Profit-sharing bonus | Remote-first work environmentSenior-level Full TimeArlington, VA (remote); Baltimore, MD (remote); … R5d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Dashboarding | Data Modeling | Data integration | LookMLCompetitive benefits | Global team collaboration | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeAustin, TX (remote); Dallas, TX (remote); … R5d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Affiliate Marketing Analytics | Business Intelligence | Data Ingestion | Data LiteracyCareer growth | Competitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeChicago, IL (remote); Columbus, OH (remote); … R5d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | CAC | Dashboard Development | Data Ingestion | Data ModelingSenior-level Full TimeDenver, CO (remote); Las Vegas, NV … R5d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Customer Acquisition | Customer Acquisition Cost | Dashboarding | Data ModelingCompetitive benefits | Professional growth | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeLos Angeles, CA (remote); Portland, OR … R5d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | CAC | Dashboard Development | Data Literacy | Data ModelingEducation and best practices | Remote-first team | Self service analytics enablementSenior-level Full TimeAsheville, NC (remote); Atlanta, GA (remote); … R5d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Customer Acquisition | Customer Acquisition Cost | Customer Acquisition Cost CAC | DashboardingCompetitive benefits | Profit-sharing bonus | Remote-first work cultureSenior-level Full TimeArlington, VA (remote); Baltimore, MD (remote); … R5d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Dashboard Development | Data Ingestion | Data Modeling | Data StandardizationCompetitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeAustin, TX (remote); Dallas, TX (remote); … R5d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | CAC Analytics | Creative analysis | Dashboarding | Data GovernanceEducation and best practices | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeChicago, IL (remote); Columbus, OH (remote); … R5d ago
-
Lead Analytics Manager USD 120K-150KDashboard Development | Data Ingestion | Data Modeling | Data Quality | Data StandardizationCompetitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeDenver, CO (remote); Las Vegas, NV … R5d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Creative analysis | Data Literacy | Data Modeling | Data StandardizationEducation and best practices | Performance-based culture | Remote-first teamSenior-level Full TimeLos Angeles, CA (remote); Portland, OR … R5d ago
-
Senior Full Stack AI Developer USD 105K-231KAWS | Authentication | Azure | CI/CD | CSSFlexible time off | Learning resourcesSenior-level Full Time999 REMOTE, United States R5d ago
-
Database Developer USD 100K-120KData Architecture | Data Lake | Data Modeling | Data Visualization | Data WarehouseMid-level Full TimeRemote, XX, United States R6d ago
-
AI Developer - Model Creation & Full Stack USD 130K-145KAWS | Angular | Azure Databricks | CI/CD | Cloud platformRemote workMid-level Full TimeWork from home, VA, United States R6d ago
-
Hadoop Big Data Developer USD 100K-150KAWS EMR | Airflow | Apache Atlas | Apache Hive | Apache HudiRemote workSenior-level Full TimeUnited States - Remote R6d ago