Especialista de Engenharia de Dados II – Tech Lead
Tasks
- Build real time ingestion with Spark Structured Streaming
- Conduct technical code and architecture reviews
- Create Data Mesh standards and best practices
- Define lakehouse architecture
- Design batch and streaming pipelines architectures
- Develop scalable pipelines in Python Scala SQL and Java
- Ensure data quality
- Ensure scalability security governance and performance
- Establish data observability monitoring alerts metrics
- Facilitate discovery and design for data products
- Implement AWS Lake Formation governance
- Implement Delta Live Tables
- Lead and mentor data engineers
- Lead internal workshops and technical communities
- Manage data access with AWS IAM
- Set up Kafka and Kinesis ingestion
- Support CI CD and versioning for pipelines
- Translate business needs into scalable data products
- Use Unity Catalog for data governance
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | AWS Glue | AWS IAM | AWS Lake Formation | AWS Lambda | Alerting | Amazon Kinesis | Amazon S3 | Apache Spark | CI/CD | Data Governance | Data Observability | Data Quality | Data mesh | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Java | Kafka | Lake Formation | Lakehouse | Metrics | Monitoring | Python | SQL | Scala | Structured Streaming | Unity Catalog | Version control
Education
N/A
Roles
Architect | Data Engineer | Data Platform | Data Platform Architect | Engineer | Lead | Platform Architect | Tech Lead
Related jobs
-
AI Data Engineer BRL 150K-240KAWS | AWS EventBridge | AWS Lambda | AWS S3 | Automated testingDEI initiatives opportunities | Fully remote or hybrid work | Healthcare stipend | National holidays | Paid mental health recharge daysMid-level Full TimeRemote - Brazil R15h ago
-
Amazon EMR | Apache Airflow | Apache Kafka | Big Data | Data ModelingCareer development opportunities | Dental insurance | Flexible schedule | Health insurance | Home office supportSenior-level Full TimeBrazil23h ago
-
Senior Analytics Engineer BRL 78K-108KA/B | A/B Testing | Apache Airflow | B testing | BigQueryRemote workSenior-level Full TimeBrasil R1d ago
-
AWS DMS | AWS Lambda | Airflow | CDC | CI/CDChildcare support | Dental insurance | Extended parental leave | Health insurance | Learning platformSenior-level Full TimeBrazil1d ago
-
A/B | A/B Testing | API Gateway | AWS | AWS LambdaChildcare assistance | Dental insurance | Discount club | Employee profit-sharing | Gym benefitsSenior-level Full TimeBrazil1d ago
-
Pessoa Engenheira de Dados Pleno BRL 96K-119KApache Airflow | Apache Spark | BigQuery | CI/CD | Cloud StorageMid-level Full TimeSão Paulo1d ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Lakes | Data WarehousingFlexible schedule | Mentorship | Personalized growth roadmaps | Remote and office options | TechtalksMid-level Full TimeSorocaba, Brazil1d ago
-
Algorithms | Amazon Redshift | Amazon Web Services | DBT | Data LakeFlexible schedule | Personalized growth roadmap | Professional mentorship | Remote work options | TechtalksMid-level Full TimePorto Alegre, Brazil1d ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Architecture | Data LakeEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimeRecife, Brazil1d ago
-
AWS | Amazon Redshift | Amazon Web Services | DBT | Data LakesFlexible schedule | Mentorship | Office option | Remote work option | TechtalksMid-level Full TimeSalvador, Brazil1d ago
-
AWS | Amazon Redshift | DBT | Data Engineering | Data LakesEducation budget | Fitness budget | Flexible schedule | Mentorship | Personalized growth roadmapsMid-level Full TimeCuritiba, Brazil1d ago
-
AWS | Amazon Redshift | DBT | Data Lakes | Data ModelingEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionMid-level Full TimeSão Bernardo do Campo, Brazil1d ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Lakes | Data QualityFitness budget | Flexible schedule | Mentorship | Remote work options | Team activity budgetMid-level Full TimeFlorianópolis, Brazil1d ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Lakes | Data StructuresEducation budget | Fitness budget | Flextime | Mentorship | Personalized growth roadmapsMid-level Full TimeCampinas, Brazil1d ago
-
AWS | Algorithms | Amazon Redshift | Amazon Web Services | Computer ScienceEducation budget | Fitness budget | Flextime | Mentorship | Office optionsMid-level Full TimeRio de Janeiro, Brazil1d ago
-
AWS | Amazon Redshift | DBT | Data Lakes | Data WarehousingEducation budget | Fitness budget | Flextime | Mentorship | Office optionsMid-level Full TimeBelo Horizonte, Brazil1d ago
-
AWS | Algorithms | Amazon Redshift | DBT | Data LakesEducation budget | Fitness budget | Flextime | Mentorship | Office optionsMid-level Full TimeBrasília, Brazil1d ago
-
Algorithms | Amazon Redshift | Amazon Web Services | DBT | Data LakesEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimeSão Paulo, Brazil1d ago
-
[Job-28865] Engenheiro de Dados Senior, Brazil BRL 18K-18KACID | Azure | Azure Data | Azure Data Factory | Azure DatabricksChildcare support | Continuous learning platform | Dental insurance | Discounts | Extended parental leaveSenior-level Full TimeBrazil1d ago
-
AWS Athena | AWS Glue | AWS S3 | Amazon Redshift | Data ArchitectureHybrid workSenior-level Full TimeSão Paulo, Brasil1d ago
-
Engenheiro de Dados Sênior BRL 58K-72KAI | AWS | AWS Glue | Agile | Amazon AthenaTravel opportunitiesSenior-level Full TimeSão Carlos, SP, Brazil1d ago
-
Entry-level Full TimeLondrina1d ago
-
Analista de Dados Sênior (AWS) - 132464 BRL 96K-119KAWS | AWS Glue | AWS Lambda | AWS Step Functions | Amazon AthenaMid-level Full TimeAlphaville - Barueri, BR, 06.454-0002d ago
-
Principal Data Engineer Team Lead - OP02110 BRL 271K-370KAWS | Apache Airflow | Code Reviews | DBT | DagsterEnglish lessons | Health insurance | Paid sick days | Paid time off | Remote workSenior-level Full TimeSão Paulo, São Paulo, Brazil - … R2d ago
-
Consultor de Dados - Business Intelligence BRL 90K-102KArtificial Intelligence | DAX | Data Modeling | Data Pipelines | ETLOnline assessmentMid-level Full TimeBelo Horizonte, Brazil2d ago