Especialista de Engenharia de Dados II – Tech Lead
Tasks
- Build real time ingestion with Spark Structured Streaming
- Conduct technical code and architecture reviews
- Create Data Mesh standards and best practices
- Define lakehouse architecture
- Design batch and streaming pipelines architectures
- Develop scalable pipelines in Python Scala SQL and Java
- Ensure data quality
- Ensure scalability security governance and performance
- Establish data observability monitoring alerts metrics
- Facilitate discovery and design for data products
- Implement AWS Lake Formation governance
- Implement Delta Live Tables
- Lead and mentor data engineers
- Lead internal workshops and technical communities
- Manage data access with AWS IAM
- Set up Kafka and Kinesis ingestion
- Support CI CD and versioning for pipelines
- Translate business needs into scalable data products
- Use Unity Catalog for data governance
Perks/Benefits
- N/A
Skills/Tech-stack
AWS | AWS Glue | AWS IAM | AWS Lake Formation | AWS Lambda | Alerting | Amazon Kinesis | Amazon S3 | Apache Spark | CI/CD | Data Governance | Data Observability | Data Quality | Data mesh | Databricks | Delta Lake | Delta Live Tables | Delta Live) | Java | Kafka | Lake Formation | Lakehouse | Metrics | Monitoring | Python | SQL | Scala | Structured Streaming | Unity Catalog | Version control
Education
N/A
Roles
Architect | Data Engineer | Data Platform | Data Platform Architect | Engineer | Lead | Platform Architect | Tech Lead
Related jobs
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | Amazon AthenaEducation budget | Fitness budget | Flexible schedule | Mentorship | Remote and office optionsMid-level Full TimeCampinas, Brazil11h ago
-
AWS | AWS Systems Manager | AWS systems | Amazon RDS | AthenaEducation budget | Fitness budget | Flextime | Mentorship | Office work optionsMid-level Full TimeCuritiba, Brazil11h ago
-
AWS | AWS CloudWatch | Amazon RDS | Athena | AutomationEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimeSorocaba, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS SAMEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimeSão Paulo, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS SAMEducation budget | Exciting projects | Fitness budget | Flextime | MentorshipMid-level Full TimeFlorianópolis, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS ServerlessFlextime | Growth roadmaps | Mentorship | Office options | Remote optionsMid-level Full TimeSalvador, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS SAMEducation budget | Fitness budget | Flexible schedule | Mentorship | Personalized growth roadmapsMid-level Full TimePorto Alegre, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS Lambda | AWS SAMFlextime | Mentorship | Office options | Personalized growth roadmaps | Remote work optionsMid-level Full TimeBrasília, Brazil11h ago
-
AWS | AWS CloudFormation | AWS Glue | AWS SAM | Amazon AthenaEducation budget | Fitness budget | Flexible schedule | Mentorship | Office optionsMid-level Full TimeSão Bernardo do Campo, Brazil11h ago
-
AWS | AWS SAM | Amazon RDS | Athena | AutomationFlextime | Mentorship | Personalized growth roadmaps | Remote work options | TechtalksMid-level Full TimeRecife, Brazil11h ago
-
AWS | Amazon Redshift | DBT | Data Architecture | Data LakesFlextime | Growth roadmaps | Mentorship | Office options | Remote optionsMid-level Full TimeSão Paulo, Brazil11h ago
-
AWS | Algorithms | Amazon Redshift | DBT | Data LakeFlextime | Growth roadmaps | Mentorship | Office work options | Remote workMid-level Full TimePorto Alegre, Brazil11h ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Lakes | Data StructuresFlextime | Mentorship | Personalized growth roadmaps | Remote work option | TechtalksMid-level Full TimeSão Bernardo do Campo, Brazil11h ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Architecture | Data LakesFlextime | Mentorship | Office options | Personalized growth roadmaps | Remote work optionsMid-level Full TimeBrasília, Brazil11h ago
-
AWS | Algorithms | Amazon Redshift | Computer Science | Computer science fundamentalsEducation budget | Fitness budget | Flextime | Mentorship | Personalized growth roadmapsMid-level Full TimeRecife, Brazil11h ago
-
AWS | Algorithms | Amazon Redshift | Amazon Web Services | DBTEducation budget | Fitness budget | Flextime | Growth roadmaps | MentorshipMid-level Full TimeCampinas, Brazil11h ago
-
AWS | Amazon Redshift | DBT | Data Architecture | Data LakesFlextime | Growth roadmaps | Mentorship | Remote work options | TechtalksMid-level Full TimeRio de Janeiro, Brazil11h ago
-
AWS | Amazon Redshift | DBT | Data Architecture | Data LakehouseEducation budget | Fitness budget | Flextime | Mentorship | Office optionsMid-level Full TimeFlorianópolis, Brazil11h ago
-
AWS | Algorithms | Amazon Redshift | DBT | Data LakesFlexible schedule | Mentorship | Personalized growth roadmaps | Remote work options | TechtalksMid-level Full TimeCuritiba, Brazil11h ago
-
Amazon Redshift | Amazon Web Services | DBT | Data Architecture | Data LakesEducation budget | Fitness budget | Flexible schedule | Flextime | MentorshipMid-level Full TimeSalvador, Brazil11h ago
-
AWS | Amazon Redshift | DBT | Data Lakes | Data WarehousingFlextime | Mentorship | Office options | Personalized growth roadmaps | Remote workMid-level Full TimeSorocaba, Brazil11h ago
-
AWS | Algorithms | Amazon Redshift | Amazon Web Services | DBTFlexible schedule | Mentorship | Office options | Personalized growth roadmaps | Remote optionsMid-level Full TimeBelo Horizonte, Brazil11h ago
-
Senior Data Management Professional - Data Engineering - Company Financials Market Data BRL 234K-275KArtificial Intelligence | CSV | Data Architecture | Data Modeling | Data QualitySenior-level Full TimeSao Paulo12h ago
-
Senior-level Full TimeBrasil13h ago
-
Bash | Cloud infrastructure | Data Processing | Docker | GCPMid-level Full TimeSão Paulo, Brazil19h ago