Apache Spark Developer
Tasks
- Build Spark transformations with PySpark Scala and Spark SQL
- Deploy Spark workloads on Databricks EMR Azure Synapse and Kubernetes
- Design distributed data processing applications using Apache Spark
- Develop batch and real time ETL ELT pipelines
- Develop reusable Spark libraries and metadata driven ingestion pipelines
- Implement data quality validation reconciliation monitoring and error handling
- Integrate Spark with data warehouses lakehouses and reporting platforms
- Modernize legacy ETL workloads into Spark based architectures
- Optimize spark jobs for performance and resource utilization
- Process structured semi structured and streaming data sources
- Troubleshoot production distributed processing and cluster issues
Perks/Benefits
- N/A
Skills/Tech-stack
ARM Templates | AWS Glue | AWS Step Functions | Agile Scrum | Amazon EMR | Amazon S3 | Apache Airflow | Apache Flink | Apache Hudi | Apache Iceberg | Apache Kafka | Apache NiFi | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure DevOps | Azure Resource | Azure Resource Manager | Azure Synapse | Azure Synapse Analytics | Broadcast Join | CI/CD | Caching | Catalyst Optimizer | Cloud Dataproc | Cloud Storage | Control-M | DAG | Data Factory | Data Lake | Data Lake Architecture | Data Lake Storage Gen2 | Data Warehousing | DataFrame | DataSet | Databricks | Datadog | Deequ | Delta Lake | Dimensional Modeling | Docker | Event Hubs | Fault Tolerance | Feature Engineering | Git | GitHub Actions | Google Cloud | Google Cloud Dataproc | Google Cloud Storage | Grafana | Great Expectations | HDFS | Hadoop | Hive | Jenkins | Kubernetes | MLflow | OpenTelemetry | Oracle | Parquet | Partitioning | PostgreSQL | Presto | Prometheus | PySpark | RDD | Resource manager | SQL | SQL Server | Scala | Shuffling | Snowflake | Spark MLlib | Spark SQL | Step Functions | Structured Streaming | Synapse Analytics | Teradata | Terraform | Trino | Tungsten engine | YARN
Education
N/A
Roles
Related jobs
-
Software Engineer, AI/ML USD 102K-160KAWS | AWS Bedrock | Azure | Azure OpenAI | C#401k matching | Disability insurance | Employee ownership | Growth opportunities | Health insuranceSenior-level Full TimeUSA-GA-Office-Atlanta, United States R2d ago
-
Lead AI Engineer - Remote USD 145K-249KAWS Bedrock | Agent Orchestration | Agentic AI | Docker | FastAPI401k contribution | Comprehensive benefits package | Remote work within the United StatesSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R2d ago
-
Senior Power BI Developer USD 91K-163KAccess Control | Active Directory | Azure Active Directory | Azure Data | Azure Data Lake401k contribution | Career development opportunities | Comprehensive benefits package | Equity stock purchase | Incentive and recognition programsSenior-level Full TimeEden Prairie, Minnesota, UNITED STATES; REMOTE R2d ago
-
ETL Developer USD 115K-175KAWS | AWS Glue | Apache Airflow | Auditability | AutosysCareer growth potential | Remote workSenior-level Full TimeUnited States - Remote R2d ago
-
Sr. Optimizely Developer USD 130K-175K.Net Core | A/B | A/B Testing | API Integration | Adobe CommerceSenior-level Full TimeRemote, MA, US R3d ago
-
AI Solutions Developer USD 113K-170KAPI Development | AWS | Artificial Intelligence | Backend Development | Cloud Computing401k matching | Adoption Assistance | FSA | Fertility assistance | HSAEntry-level Full TimeCHICAGO, United States R3d ago
-
AI Solutions Developer USD 113K-170KAPI Development | AWS | Backend Development | Cloud Computing | Data Governance401k employer match | Adoption and fertility assistance | Background check required | Choice time off | FSA and HSA accountsEntry-level Full TimeCHICAGO, United States R3d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Creative Analytics | Customer Acquisition | Customer Acquisition Cost | DashboardingCareer growth path | Education and best practice support | Remote-first culture | Self service analytics enablementSenior-level Full TimeAsheville, NC (remote); Atlanta, GA (remote); … R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAd-Hoc | Ad-hoc analysis | Affiliate Marketing | Dashboard Development | Data ModelingCareer growth path | Competitive benefits | Profit-sharing bonus | Remote-first work environmentSenior-level Full TimeArlington, VA (remote); Baltimore, MD (remote); … R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Dashboarding | Data Modeling | Data integration | LookMLCompetitive benefits | Global team collaboration | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeAustin, TX (remote); Dallas, TX (remote); … R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Affiliate Marketing Analytics | Business Intelligence | Data Ingestion | Data LiteracyCareer growth | Competitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeChicago, IL (remote); Columbus, OH (remote); … R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | CAC | Dashboard Development | Data Ingestion | Data ModelingSenior-level Full TimeDenver, CO (remote); Las Vegas, NV … R4d ago
-
Lead Data Analyst, Marketing USD 120K-150KAffiliate Marketing | Customer Acquisition | Customer Acquisition Cost | Dashboarding | Data ModelingCompetitive benefits | Professional growth | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeLos Angeles, CA (remote); Portland, OR … R4d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | CAC | Dashboard Development | Data Literacy | Data ModelingEducation and best practices | Remote-first team | Self service analytics enablementSenior-level Full TimeAsheville, NC (remote); Atlanta, GA (remote); … R4d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Customer Acquisition | Customer Acquisition Cost | Customer Acquisition Cost CAC | DashboardingCompetitive benefits | Profit-sharing bonus | Remote-first work cultureSenior-level Full TimeArlington, VA (remote); Baltimore, MD (remote); … R4d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Dashboard Development | Data Ingestion | Data Modeling | Data StandardizationCompetitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeAustin, TX (remote); Dallas, TX (remote); … R4d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | CAC Analytics | Creative analysis | Dashboarding | Data GovernanceEducation and best practices | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeChicago, IL (remote); Columbus, OH (remote); … R4d ago
-
Lead Analytics Manager USD 120K-150KDashboard Development | Data Ingestion | Data Modeling | Data Quality | Data StandardizationCompetitive benefits | Profit-sharing bonus | Remote-first cultureSenior-level Full TimeDenver, CO (remote); Las Vegas, NV … R4d ago
-
Lead Analytics Manager USD 120K-150KAffiliate Marketing | Creative analysis | Data Literacy | Data Modeling | Data StandardizationEducation and best practices | Performance-based culture | Remote-first teamSenior-level Full TimeLos Angeles, CA (remote); Portland, OR … R4d ago
-
Senior Full Stack AI Developer USD 105K-231KAWS | Authentication | Azure | CI/CD | CSSFlexible time off | Learning resourcesSenior-level Full Time999 REMOTE, United States R4d ago
-
Database Developer USD 100K-120KData Architecture | Data Lake | Data Modeling | Data Visualization | Data WarehouseMid-level Full TimeRemote, XX, United States R4d ago
-
AI Developer - Model Creation & Full Stack USD 130K-145KAWS | Angular | Azure Databricks | CI/CD | Cloud platformRemote workMid-level Full TimeWork from home, VA, United States R4d ago
-
Hadoop Big Data Developer USD 100K-150KAWS EMR | Airflow | Apache Atlas | Apache Hive | Apache HudiRemote workSenior-level Full TimeUnited States - Remote R5d ago
-
AWS | Azure | Cognos Analytics | ETL | Enterprise Performance401k match | Dental insurance | Employee assistance program | Flexible spending account | Health insuranceSenior-level Full TimeIrving, TX, US R5d ago
-
Hadoop Big Data Developer USD 86K-103KAWS EMR | Airflow | Apache Atlas | Apache Hudi | Apache IcebergRemote workSenior-level Full TimeUnited States - Remote R5d ago