Apache Spark Developer
Tasks
- Build scalable batch and real time ETL ELT pipelines
- Deploy Spark workloads on Databricks EMR Azure Synapse and Kubernetes
- Design distributed data processing applications using Apache Spark
- Develop Spark applications using PySpark Scala and Spark SQL
- Develop reusable Spark libraries and metadata driven ingestion pipelines
- Implement data quality validation monitoring and automated error handling
- Integrate Spark with data warehouses lakehouses and reporting platforms
- Modernize legacy ETL workloads into Spark based architectures
- Optimize Spark job performance using partitioning and shuffle tuning
- Process structured semi structured and streaming data
- Troubleshoot production cluster and data quality issues
Perks/Benefits
- N/A
Skills/Tech-stack
ARM Templates | AWS EMR | AWS Glue | Agile Scrum | Amazon S3 | Apache Airflow | Apache Hadoop | Apache Hudi | Apache Iceberg | Apache Kafka | Apache Spark | Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2 | Azure DevOps | Azure Synapse | Azure Synapse Analytics | CI/CD | Cloud Dataproc | Cloud Storage | Data Factory | Data Lake Storage Gen2 | Databricks | Datadog | Delta Lake | Docker | Event Hubs | Git | GitHub Actions | Google Cloud | Google Cloud Dataproc | Google Cloud Storage | Grafana | HDFS | Hive | Infrastructure as Code | Jenkins | Kafka | Kubernetes | OpenTelemetry | Oracle | Parquet | PostgreSQL | Prometheus | PySpark | SQL | SQL Server | Scala | Snowflake | Spark SQL | Spark Structured Streaming | Structured Streaming | Synapse Analytics | Teradata | Terraform | YARN | “as-code”
Education
N/A
Roles
Related jobs
-
Change Data Capture | Cloud Data Integration | Data Capture | Data Integration Patterns | Data WarehouseRemote workSenior-level Full TimeAustin, TX R1d ago
-
Principal AI Developer USD 254K-382KAWS | Agentic Systems | Alert triage | Apache Flink | Apache KafkaComprehensive private benefits plan | Employee Assistance Program (EAP) | Fertility support | Flexible time off | Medical dental disability life AD and DSenior-level Full TimeRemote - USA - New York, … R1d ago
-
Senior AI Software Developer USD 107K-195KAWS | Agile | Automated testing | Azure | C++100 percent remoteSenior-level Full Time6314 Remote/Teleworker US, United States R1d ago
-
C++ | Calibration | Derivatives | Intraday Risk | JSONEmployee resource groups | Family building benefits | Generous vacation | Health and wellbeing benefits | Hybrid work modelExecutive-level Full TimeNew York, 1 Columbus Circle, United … R1d ago
-
Senior Business Intelligence Developer - Tableau USD 83K-142KBig Data | CSS | Calculated Fields | D3.js | Data Load ScriptsSenior-level Full TimeRemote TX Frisco 75035, United States R1d ago
-
Data Visualization Developer USD 125K-135KData Modeling | ETL | Excel | Pandas | Power Automate401k | Dental insurance | Employee assistance program | Medical insurance | Paid time offMid-level Full TimeWashington, DC, DC R1d ago
-
Intermediate Cloud / AI Developer USD 96K-130KAlerting | Application Lifecycle Management | Application lifecycle | Audit Logging | Authentication401k match | Health and wellness benefits | Paid Holidays | Paid parental leave | Paid time offMid-level Full TimeUSA PA Home Office (PAHOME), United … R1d ago
-
Associate Cloud / AI Developer USD 83K-113KAI integration | Access Control | Alerts | Audit Logging | CI/CD401k match | Health and wellness benefits | Paid Holidays | Paid family leave | Paid parental leaveMid-level Full TimeUSA PA Home Office (PAHOME), United … R1d ago
-
Senior Cloud / AI Developer USD 111K-150KAgile | Alerting | Audit Logging | Authentication | Authorization401k with company match | Dental insurance | Flex work weeks | Health insurance | Paid HolidaysSenior-level Full TimeUSA PA Home Office (PAHOME), United … R1d ago
-
Applications & Data Analytics Specialist - Sr. Lead USD 98K-151KAPIs | DAX | Dashboard Optimization | Data Lineage | Data ModelingHybrid work flexibility | Up to 25 percent travelSenior-level Full TimeNew Orleans, Louisiana, United States R1d ago
-
APIs | AWS | Amazon Web Services | Angular | AzureIn-person engagementSenior-level Full TimeSan Antonio, TX, US, 78249 R1d ago
-
API Integration | AWS | Amazon Bedrock | Angular | Application MonitoringIn-person work requirement | Secret trust clearance requiredEntry-level Full TimeSan Antonio, TX, US, 78249 R1d ago
-
Hadoop Solutions Developer USD 100K-150KAWS EMR | Airflow | Apache Atlas | Apache Flink | Apache HiveSenior-level Full TimeUnited States - Remote R1d ago
-
Senior PowerBI Consultant USD 107K-140KAgile Scrum | Azure DevOps | Azure Fabric | BI Desktop | Calculation GroupsSenior-level Full TimeAnywhere USA, United States R2d ago
-
Collibra Engineer / Data Governance USD 140K-155KBPMN | Collibra | Data Catalog | Data Governance | Data Monitoring401k | Dental | Disability insurance | Employee assistance program | Employee discountsMid-level Full TimeWashington, D.C., DC, US R2d ago
-
Analysis Services | BI Desktop | DAX | Data Warehousing | Dimension tablesSenior-level Full TimeCleveland, OH, United States R2d ago
-
Salesforce Data Architect USD 160K-170KAnypoint platform | Apex | Bulk API | CRM Analytics | CloudHubDental insurance | Federal Holidays | Health insurance | LTD insurance | Life insuranceSenior-level Full Time1600 Spring Hill Road suite 240, … R2d ago
-
Sr. Power BI & DBA USD 95K-125KAPI Automation | Azure AD | CTE | DAX | Data Gateway401k retirement plan | AD D Insurance | Dental insurance | Life insurance | Medical insuranceSenior-level Full TimeUnited States R3d ago
-
Analysis Services | Analysis and Design | Dashboard governance | Data Gateway | Data ModelSenior-level Full TimeAustin, Texas; Information Technology R3d ago
-
Python | Snowflake | Snowpipe | User Defined FunctionsHybrid work schedule | Local only candidatesSenior-level Full TimeAustin, TX R3d ago
-
Bash | C++ | CI/CD | Cloud Computing | Database Design401k | Dental insurance | Health insurance | Paid Holidays | Remote work optionsSenior-level Full TimeBethesda, Maryland R3d ago
-
AI Software Engineer USD 110K-150KAsynchronous processing | Azure | CI/CD | Docker | EmbeddingsConnectivity reimbursement | Equipment provided | Private workspace requirement | Work from homeMid-level Full TimeUSA - IL (Remote), United States R3d ago
-
AI Software Engineer USD 110K-150KAsynchronous processing | Azure | CI/CD | Docker | EmbeddingsConnectivity reimbursement | Work from homeMid-level Full TimeUSA - IL (Remote), United States R3d ago
-
Data Analyst III USD 120K-140KAWS | Agile | BI Dashboard | BI Dashboard Development | Business Intelligence401k plan | Employee assistance program | Paid federal holidays | Paid time off | Remote work possibilitySenior-level Full TimeOrlando, Florida, United States - Remote R3d ago
-
Senior-level Full TimeUnited States - Remote R3d ago