Data Engineer (Spark)
Tasks
- Collaborate with teams on data requirements
- Design data pipelines for streaming and batch
- Develop and maintain data processing platform
- Leverage AWS for infrastructure scaling
- Manage data lake using Iceberg
- Monitor and troubleshoot platform performance and accuracy
- Optimize data workflows for ingestion processing storage
- Write and maintain Python code for data processing
Perks/Benefits
- Career development
- Conference support
- Flexible work arrangements
- Integration budget
- Knowledge sharing
- Language classes
- Medical coverage
- Paid time off
- Sponsored training
- Sports packages
- Team building events
- Wellness support
- Work equipment provided
- Work remote options
Skills/Tech-stack
AWS | Apache Airflow | Apache Iceberg | Apache Kafka | Apache NiFi | Apache Spark | Cloudera | DBT | Data Governance | Data Management | Data Modeling | Databricks | Dimensional Modeling | Docker | Google BigQuery | Java | Kubeflow | Looker | MLOps | MLflow | Python | Scala
Education
Roles
Related jobs
-
Senior Backend Software Developer, Platform & Services CAD 123K-161KAPI Design | AWS | ClickHouse | Data pipeline | Database DesignDental insurance | Flexible work hours | Health insurance | Open PTO policy | Parental leave top-upSenior-level Full TimeCanada (Remote) R15h ago
-
Senior Data Engineer CAD 132K-208KAccess Management | Cassandra | Data Governance | Data Modeling | Data QualitySenior-level Full TimeRemote - Toronto, Ontario, Canada R17h ago
-
NumPy | Pandas | Python | SciPy | Scikit-learnFlexible schedule | Freelance projects | Part-time engagement | Stable internet requiredSenior-level FreelanceUnited States - Remote R18h ago
-
NumPy | Pandas | Probability | Python | SciPyFlexible project participation | Freelance project-based work | Part-time hours | Stable internet requiredMid-level FreelanceItaly - Remote R18h ago
-
NumPy | Pandas | Python | SciPy | Scikit-learnFlexible schedule | Freelance project-based work | Stable internet connection requiredMid-level FreelancePortugal - Remote R18h ago
-
NumPy | Pandas | Python | SciPy | Scikit-learnFlexible schedule | Freelance project-based work | Part-time work | Remote workSenior-level FreelanceUnited Kingdom - Remote R18h ago
-
Mathematical Statistics | NumPy | Pandas | Probability theory | PythonSenior-level FreelanceRomania - Remote R18h ago
-
Applied statistics | Mathematical Statistics | NumPy | Numerical analysis | PandasFlexible schedule | Freelance projects | Remote work | Stable internet requiredSenior-level FreelanceSingapore - Remote R18h ago
-
Sr. Engineer - Data Analytics (Hybrid) USD 140K-215KAmazon Web Services | Apache Airflow | Apache Kafka | CQL | CassandraCompetitive vacation and holidays | Employee networks and volunteer opportunities | Employee wellness programs | Hybrid work | Paid adoption leaveSenior-level Full TimeUSA NY Remote, United States R18h ago
-
Senior AI Business Process Engineer CAD 120K-170KAPI Design | Agile | BPMN 2.0 | Backlog Management | Camunda 8Career development opportunitiesSenior-level Full TimeOntario, Canada - Remote R18h ago
-
Senior AI Business Process Engineer USD 128K-182KAPI | Agile | BPMN 2.0 | Backlog Management | Camunda 8Career development opportunities | Equal employment opportunitySenior-level Full TimeDallas, Texas, United States - Remote R18h ago
-
API Design | Distributed Systems | Kubernetes | Microservices | PostgreSQLDaily catered lunch | Medical, dental & vision coverage | Unlimited PTO | Visa supportSenior-level Full TimeSan Francisco, CA; Hybrid R18h ago
-
Senior AI Engineer INR 2500K-3200KAgile | Amazon Web Services | CSS | Confluence | GitHome office setup reimbursement | Medical benefits | Mental wellness support | Paid time off | Parental leaveSenior-level Full TimeChennai or Remote, India R1d ago
-
Data Scientist II - Computer Vision USD 140K-170KComputer Vision | Convolutional Neural Networks | Deep learning | Experiment tracking | Field extractionMid-level Full TimeRemote - US R1d ago
-
Amazon SageMaker | Apache Airflow | CI/CD | Distributed Systems | DockerCross-functional collaboration | Remote workSenior-level Full TimePorto, Portugal R1d ago
-
AI Software Developer USD 113K-188KAWS | Agentic AI | Algorithms | Azure | Cloud infrastructure401k employer match | Employer Paid Short Term Disability Long Term Disability | Employer-paid life insurance | Equity incentive plan | Federal Holiday Paid LeaveSenior-level Full TimeAlexandria, Virginia, United States - Remote R1d ago
-
Senior-level Full TimeJakarta, Indonesia; Chennai, India; Remote (India) R1d ago
-
Applied AI Engineer CAD 115K-145KAI orchestration | API Development | AWS | AWS Bedrock | Anthropic Claude401k match | Annual professional development budget | Charitable donation match | Commuter benefits | Flexible time offMid-level Full TimeRemote - Ontario, Canada R1d ago
-
AI Research Engineer USD 300K-425KApplied cryptography | Artificial Intelligence | Blockchain | Cryptographic mechanisms | Distributed SystemsEntry-level Full TimeAnywhere R1d ago
-
Software Engineer, Backend - Remote (US or International) INR 1500K-2000KApache Cassandra | Apache Druid | Apache Hadoop | Apache Kafka | Apache PinotOpen Source Community Collaboration | Remote workEntry-level Full TimeRemote (India) R1d ago
-
Senior Data Engineer EUR 54K-90KAgile | Azure Data | Azure Data Factory | Azure Data Lake | Azure DevOpsAdditional health insurance | Compensated certificates | Flexible time off | Language lessons | Learning lunchesSenior-level Full TimeRemote job R1d ago
-
Senior Software Engineer (Golang/Python) USD 143K-180KAWS | CI/CD | Docker | Git | GoFlexible collaboration | Remote workSenior-level Contract Full TimeLatin America R1d ago
-
Data Engineer (East Coast Remote) USD 100K-138KApache Airflow | Cloud Composer | Cloud Storage | DBT | DBT CloudMid-level Full TimeUnited States R1d ago
-
Senior AI Engineer - Agentic AI Platform USD 117K-220KAI Foundry | Agent Framework | Agent systems | Agentic AI | Azure AIConference attendance support | Flexible working hours | Inclusion and diversity focus | Leadership influence | Paid training budgetMid-level Full TimeRemote job R1d ago
-
AI Engineer II - Agentic AI Platform USD 130K-242KAI Search | Azure AI | Azure AI Search | Azure OpenAI | Azure OpenAI ServiceAI training budget | Certification support | Conference attendance | Diversity initiatives | Flexible working hoursSenior-level Full TimeRemote job R1d ago