Lead Spark Data Engineer
Tasks
- Architect data solutions on Spark and Flink
- Build Spark Adapter
- Develop Query Validator
- Document solutions and workflows
- Enforce data governance, security, and quality standards
- Ensure parity between Spark and Flink implementations
- Implement IoT Query Language grammar
- Implement relationships logic to optimize graph traversal
- Monitor and tune workloads for performance
- Optimize Azure and Databricks resources
- Participate in agile activities
- Transform and clean data using SQL, Python, Java
Perks/Benefits
- N/A
Skills/Tech-stack
ANTLR v4 | ASTs | Apache Spark | Azure DevOps | Azure ecosystem | CI/CD | DSLS | Data Governance | Data Modeling | Data Quality | Data Security | Data Warehousing | Databricks | Delta Lake | GitHub | Graph/Ontology | Java | Python | SQL
Education
Roles
Related jobs
-
Senior Data Platform Engineer (Python/AWS) - Finance BRL 258K-360KAWS CDK | AWS Glue | AWS Lambda | AWS Step Functions | Amazon ECSPaid time off | Remote work | Work autonomy | Work with top US companiesSenior-level Full TimeSão Paulo R1d ago
-
Master Data Engineer BRL 216K-275KData Cleansing | Data Integrity | Data Management | Data Quality | Data pipelineFlexible working hours | Laptop provision | Mentoring | Online courses | Referral bonusSenior-level Full TimeSão Paulo, São Paulo, Brazil - … R1d ago
-
AI & LLM Engineering Lead BRL 305K-396KAPI Design | AWS | Azure | CI/CD | Cloud infrastructureContinuous learning culture | Mentorship | Remote workSenior-level Full TimeRemote, Brazil R1d ago
-
AI & LLM Engineering Lead BRL 305K-396KAPI Design | AWS | Azure | CI/CD | Cloud platformRemote workSenior-level Full TimeRemote, Brazil R1d ago
-
Apache Airflow | Apache Hive | Apache Spark | Cause analysis | Data ModelingPaid time off | Remote work | Work autonomy | Work with top companiesSenior-level Full TimeSão Paulo R2d ago
-
Mid Software Engineer BRL 304K-396KAPI Design | Batch Processing | BigQuery | Data Enrichment | Data GovernanceMid-level Full TimeBrazil Sao Paulo - Remote Office R2d ago
-
Senior Data Engineer - Technology BRL 304K-370KAWS Glue | Agile | Amazon Athena | Amazon QuickSight | Amazon S3Autonomy | Career growth | Paid time off | Remote work | Work with top American companiesSenior-level Full TimeSão Paulo R3d ago
-
Bash | C# | C++ | Coverage.py | DockerFlexible schedule | Project-based collaboration | RemoteSenior-level Full TimeBrazil - Remote R3d ago
-
Data Analysis | Data Ingestion | Data cleaning | Exploratory Data Analysis | LLMPart-time freelance | Remote workMid-level FreelanceBrazil - Remote R3d ago
-
BigQuery | Cloud Data | Cloud Data Warehouses | Data Architecture | Data GovernanceDiversity and inclusion initiatives | Well-being benefitsSenior-level Full TimeFlorianópolis R4d ago
-
AI Engineer I (Remote) USD 30K-39KAI systems | AI systems design | AWS | CI/CD | ContainerizationProfessional growth opportunities | Remote workEntry-level Full TimeBrazil, Remote R4d ago
-
AWS | Akamai | Ansible | CI/CD | CloudFormationCareer advancement | Collaborative culture | Professional developmentMid-level Full TimeFoz do Iguaçu, Brazil (Hybrid) R4d ago
-
Bash | C# | C++ | Coverage | DockerFlexible hours | Project-based collaboration | Remote workSenior-level Full TimeBrazil - Remote R4d ago
-
Computational Algorithms | Mathematical Methods | NumPy | Pandas | PythonFlexible schedule | Professional development opportunities | Remote workMid-level FreelanceBrazil - Remote R4d ago
-
AI GTM Engineer BRL 234K-300KAI Models | API Management | JavaScript | Prompt engineering | PythonCollaborative environment | Company-provided gear | Flexible paid time off | Impactful products | Professional developmentSenior-level Full TimeBrazil - Remote R4d ago
-
Team Lead - Software Engineering BRL 234K-300KAPIs | AWS | Agile | CI/CD | Data platformsFlexible hours | Professional growth opportunities | Remote workSenior-level Full TimeSão Paulo, State of São Paulo, … R6d ago
-
AI | APIs | Agentic Orchestration | Data Pipelines | Machine LearningCareer growth opportunities | Remote workSenior-level Contract Full TimeSão Paulo, State of São Paulo, … R6d ago
-
AI Native Full Stack Engineer - OPS00063 BRL 234K-300KAI Development | Cloud Platforms | Data Engineering | Debugging | JavaHealth insurance | Paid days off | Wellness allowanceSenior-level Full TimeSão Paulo, São Paulo, Brazil - … R8d ago
-
Engenheiro(a) de Dados - Remoto BRL 20K-31KAzure Data | Azure Data Factory | DB2 | Data Architecture | Data FactoryDiversity promotion | Inclusive cultureSenior-level Full TimeBrazil - Sao Paulo R9d ago
-
.NET | AI systems | API Integration | ASP.Net Core | C#Career growth opportunities | Flexible hours | Remote workSenior-level Full TimeBrazil - Remote R9d ago
-
Mid-level Full TimeBrazil - Remote R10d ago
-
API Integration | Data Pipeline Architecture | Data pipeline | Forecasting | LLMGrowth opportunities | Innovative culture | Remote workSenior-level Full TimeBrazil - Remote R10d ago
-
Principal Engineer - Data Infrastructure BRL 304K-380KAWS DMS | DBT | Data Modeling | Data Pipeline Development | Data WarehousingCollaborative culture | Flexible schedule | Remote workSenior-level Full TimeBrazil, Remote R11d ago
-
AI tools | APIs | Automation tools | ChatGPT | ClaudeFully remote work | Global compensation | High ownership | Mentorship from AI developers | Practical AI applicationsEntry-level Full TimeSão Paulo, Brazil R11d ago
-
Senior Data Engineer | Deep Individual Data team BRL 153K-180KAirflow | Big Data | Cloud Computing | Data Analysis | Data ModelingCareer growth opportunities | Flexible benefits | Flexible schedule | Inclusive culture | Paid parental leaveSenior-level Full TimeBrazil (Remote) R11d ago