Software Engineer - Data Pipelines
Israel - Office - Tel Aviv
ILS 224K-290K (estimate) Mid-level Full Time
Tasks
- Author Airflow DAGs
- Build AWS S3 SQS SNS and EMR integrations
- Build data pipelines lifecycle end to end
- Design Snowflake schemas and pipelines
- Develop distributed data processing jobs
- Ensure data observability with monitoring
- Implement data health checks and conditional validations
- Implement distributed tracing across pipelines
- Maintain pipelines for access layer threat intelligence assets detections
- Optimize PySpark jobs
- Set up CI CD staged deployments with artifact packaging
- Write automated tests using unit and integration frameworks
Perks/Benefits
Skills/Tech-stack
AWS | Amazon EMR | Amazon S3 | Amazon SNS | Amazon SQS | Apache Airflow | Asynchronous programming | CI/CD | Data Processing | Distributed Monitoring | Distributed data | Distributed data processing | Distributed tracing | Great Expectations | PySpark | Pydantic | Pytest | Python | Snowflake | Snowpark
Education
N/A
Roles
Related jobs
-
Machine Learning Performance Engineer, Annapurna Labs ILS 139K-180KC++ | Co-design | Compiler optimization | Hardware-Software Co-design | Hardware/softwareCareer growth | Flexible work hours | Inclusive team culture | Knowledge sharing | MentorshipMid-level Full TimeTel Aviv-Yafo, Tel Aviv, ISR11h ago
-
Algorithms | Computer Vision | Data Structures | Deep learning | Machine LearningEntry-level Full TimeRamat Gan, Israel1d ago
-
Analog electronics | Digital Signal | Digital Signal Processing | FPGA | MATLABFast paced creative environment | Interdisciplinary team collaboration | Research and development focusSenior-level Full TimeRaanana, Center District1d ago
-
Senior Backend Engineer (Data Platform Group) ILS 468K-540KAKS | Apache Iceberg | Azure | CI/CD | Clean CodeSenior-level Full TimeTel-Aviv1d ago
-
Senior Data Engineer ILS 444K-528KAirflow | Amazon Redshift | Apache Kafka | Apache Spark | Argo CDSenior-level Full TimeTel-Aviv1d ago
-
Adversarial Networks | Computer Vision | Dataset curation | Deep learning | Diffusion ModelsOpportunities for publications and collaboration | Real time interactive research workMid-level Full TimeTel Aviv, Israel3d ago
-
API Integration | AWS | Asynchronous processing | Batch Processing | CI/CDSenior-level ContractKiryat Ono, Israel, IL R3d ago
-
Bash | Data Processing | Docker | GCP | Infrastructure as CodeMid-level Full TimeTel Aviv, Israel3d ago
-
AI Developer Builder USD 140K-180KAWS | Agentic Systems | Anthropic Bedrock | Context pipelines | DatadogDirect input on roadmap | High ownershipSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL R3d ago
-
Connectors | Data Processing | Data Storage | Data Structures | Data Structures and AlgorithmsSenior-level Full TimeTel Aviv, Israel4d ago
-
3D Reconstruction | Computer Vision | Deep learning | Machine Learning | Motion PredictionSenior-level Full TimeRamat Gan, Israel4d ago
-
Azure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Flexibility to work from Slovenia | Fully remote | Opportunity to contribute to large scale digital transformation initiative | Remote work autonomy and flexibilitySenior-level Contract Full TimeIsrael R4d ago
-
Senior-level Full TimeTel Aviv Cloud, Israel Office (TELAVIV)4d ago
-
APIs | AWS Lambda | AWS S3 | Amazon RDS | Amazon SNSAutonomy | Collaborative team environment | Flexible work location | Fully remote | Innovation-focused environmentSenior-level Full TimeIsrael R4d ago
-
Senior-level Full TimeIsrael, Tel Aviv4d ago
-
Mid-level Full TimeTel Aviv Cloud, Israel Office (TELAVIV)4d ago
-
Senior Data Engineer USD 149K-180KAPI Architecture | Apache Airflow | Cloud Architecture | Data Ingestion | Data Ingestion PipelinesCollaborative team | Four days per week office schedule | Mission-driven workSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL4d ago
-
Senior Data Engineer USD 150K-170KAlerting | Data Processing | Data Quality | Data Warehousing | Data orchestrationSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL4d ago
-
3D processing | AWS | CNN | Computer Vision | Convolutional Neural NetworksFlexible remote days | Hybrid work | Professional growth supportMid-level Full TimeHerzliya, Israel R5d ago
-
Entry-level Full TimeTel Aviv-Yafo, Tel Aviv District, IL5d ago
-
Mid-level Full TimeHerzliya5d ago
-
AI Performance Engineering | AI integration | AI performance | AWS | Agentic WorkflowsFlexible contract engagement | High-impact environment | Knowledge sharing and technical enablement | Work flexibility across time zonesEntry-level Contract Full TimeIsrael6d ago
-
Backend Team Leader-Data Platform ILS 283K-366KAWS | Amazon MSK | Amazon S3 | Apache Flink | Apache KafkaHybrid work modelSenior-level Full TimeTel Aviv-Yafo, Tel Aviv District, Israel R6d ago
-
Mid-level Full TimeTel Aviv6d ago
-
AWS Glue | Amazon Athena | Amazon EMR | Amazon S3 | Apache SparkMid-level Full TimeTel Aviv District, Israel6d ago