AI Data Engineer III
Tasks
- Build RAG pipelines
- Build knowledge graph and relationship models
- Build retrieval evaluation frameworks
- Create embedding and vector indexing pipelines
- Create metadata taxonomies and filtered retrieval
- Design multi tenant vector architectures
- Develop hybrid retrieval with dense and sparse search
- Develop incremental indexing for real time updates
- Implement ETL and data processing
- Implement data quality, monitoring, and alerting
- Implement document processing including PDF extraction and HTML parsing
- Implement query reformulation and reranking
- Ingest knowledge from enterprise sources
- Integrate with Snowflake for data processing
- Manage pgvector storage and indexes
- Monitor and optimize vector search performance
- Orchestrate data pipeline workflows
- Track data lineage to support agent consumption
Perks/Benefits
Skills/Tech-stack
Airflow | Async Programming | BM25 | CI/CD | DBT | Dagster | Data Modeling | Data Pipelines | Data Quality | Data quality monitoring | Databricks | Dense retrieval | ELT | ETL | Elasticsearch | Embeddings | Git | GraphQL | Knowledge graphs | Metadata filtering | Neo4j | OpenSearch | PGVector | PostgreSQL | Prefect | Python | Quality monitoring | RAG | REST API | Reranking | Retrieval evaluation | SQL | Similarity Search | Snowflake | Sparse Search | Vector Databases
Education
N/A
Roles
Related jobs
-
AWS | Analytics engineering | Artificial Intelligence | Automated testing | Cloud ArchitectureChildcare assistance | Dental insurance | Discounts on financial products | Extended maternity leave | Extended paternity leaveMid-level Full TimeSão Paulo, Brasil R20h ago
-
LLMs Data Analyst | Senior (Hybrid/SP) BRL 84K-108KAWS | Anthropic | Azure OpenAI | CI/CD | Cloud platformHybrid workSenior-level Full TimeSão Paulo, Brasil R1d ago
-
Data Management | Data Management Cloud | Data Quality | IBM DB2 | Informatica Data QualityMid-level Full TimeBrasil R1d ago
-
AI Model Evaluation | AI model | AWS | Azure | Azure DataChildcare assistance | Continuous learning platform | Dental insurance | Discount club | Extended parental leaveSenior-level Full TimeBrazil R1d ago
-
Azure | CI/CD | Databricks | Git | GitHub100 percent remote | Childcare assistance | Continuous learning platform | Dedicated language learning platform | Dental insuranceMid-level Full TimeBrazil R1d ago
-
AWS Glue | AWS Glue Data Catalog | AWS Lake | AWS Lake Formation | AWS LambdaContinuous professional development | Fully remote within Brazil | Technical mentorship | Technical ownershipMid-level Full TimeBrazil R1d ago
-
Senior-level Full TimeBrazil Sao Paulo - Remote Office R1d ago
-
AWS | AWS Lambda | Amazon EventBridge | Apache Spark | Data GovernanceChildcare assistance | Dental insurance | Exclusive Financial Product Discounts | Extended maternity leave | Extended paternity leaveSenior-level Full TimeSão Paulo, Brasil R1d ago
-
AWS | Apache Spark | Azure DevOps | CI/CD | Data GovernanceChildcare assistance | Dental insurance | Education investment | Individual development plan | Life insuranceSenior-level Full TimeBrasil R1d ago
-
Engenheiro de Dados SR BRL 60K-72KAzure Data | Azure Data Factory | Azure Data Lake | Azure Data Lake Storage | Azure Data Lake Storage Gen2Senior-level Full TimeSão Paulo, Brasil R2d ago
-
BigQuery | Data Architecture | Data Modeling | Data pipeline | Database optimizationChildcare allowance | Dental insurance | Extended maternity leave | Extended parental leave | Flexible meal benefitsMid-level Full TimeBarueri, São Paulo, Brasil R2d ago
-
Azure | Azure Blob | Azure Blob Storage | Azure Data | Azure Data FactoryConference talk sponsorship | Employee leave policies | Home office fund | Learning and development budget | Medical insuranceMid-level Full TimeBrazil R2d ago
-
AI Evaluation Engineer (QA) BRL 319K-390KAWS | Azure | CI/CD | CloudFormation | Data ModelingAccess to cutting edge AI platforms | Collaborative work environment | Diverse and inclusive culture | Growth opportunities | Hybrid working modelSenior-level Full TimeSão Paulo R2d ago
-
API Development | Airflow | Amazon Redshift | ClickHouse | DBTDental insurance | Education and courses stipend | Health plan | Paid annual leave | Sports incentiveEntry-level Full TimeSão Paulo, SP R2d ago
-
Senior Data Engineer BRL 234K-295KAWS Data | AWS Glue | AWS data platform | Amazon Web Services | Apache FlinkSenior-level Full TimeSao Paulo, SP, Brazil R3d ago
-
AWS Glue | AWS IAM | AWS Lambda | AWS S3 | AWS Step FunctionsAutonomy | Continuous learning | Fully remote | MentorshipSenior-level Full TimeBrazil R3d ago
-
Analista Machine Learning/Python BRL 30K-30KData Analysis | Machine Learning | Python | SQL | StatisticsChildcare assistance | Course discounts | Dental assistance | Meal allowance | Medical assistanceEntry-level Full TimeBrasil R3d ago
-
Apache Spark | Big Data | Data Mining | Data Modeling | Data PipelinesBonus Salaries | Childcare assistance | Dental insurance | Extended parental leave | Flexible working hoursSenior-level Full TimeBrasil R3d ago
-
AWS Bedrock | Agile | Data Storytelling | Data Visualization | Deep learningCLT employment | Hybrid work option | Remote workMid-level Full TimeAlphaville - Barueri, BR, 06.454-000 R3d ago
-
Mid-level Full TimeRemote - Brazil R3d ago
-
Anthropic Claude | Application Insights | Asynchronous systems | Azure Application Insights | Azure OpenAIB2B contract arrangement | Flexible work environment | Long-term engagement | Remote workSenior-level Contract Full TimeBrazil R5d ago
-
CAS | Engenharia de Dados - Foco em Seguros BRL 18K-18KApache Spark | Data Engineering | Data Lake | Data Modeling | DataOpsChildcare support | Dental insurance | Extended parental leave | Food allowance | GympassSenior-level Full TimeBrasil R6d ago
-
Senior Data Engineer - Databricks BRL 319K-455KAWS | Apache Spark | Automation Scripting | CI/CD | Cloud SecurityEquipment and office stipend | Flexible PTO | Learning and development stipend | Paid exams and certifications | Professional development planSenior-level Full TimeBRAZIL R6d ago
-
Data Engineer (Python) |JR (Remote) BRL 23K-23KAWS Glue | Amazon Athena | Amazon EMR | Amazon Redshift | Amazon S3Remote workEntry-level Full TimeBrasil R6d ago
-
Data Engineer (Databricks) | Specialist (Remoto) BRL 108K-120KAWS Glue | Apache Airflow | Apache Kafka | Apache Spark | CassandraMid-level Full TimeBrasil R6d ago