Senior Data Engineer - Azure Databricks

Company Description

Blend is a premier AI services provider, committed to co-creating meaningful impact for its clients through the power of data science, AI, technology, and people. With a mission to fuel bold visions, Blend tackles significant challenges by seamlessly aligning human expertise with artificial intelligence. The company is dedicated to unlocking value and fostering innovation for its clients by harnessing world-class people and data-driven strategy. We believe that the power of people and AI can have a meaningful impact on your world, creating more fulfilling work and projects for our people and clients

Job Description

• Extensive experience in guiding technical teams and delivering data-driven applications using microservice architecture.

• Strong background in Python and FastAPI, with demonstrated expertise in microservices principles, including scalability, resilience, service discovery, API gateways, and robust error handling..

• Proficient in containerization technologies (Docker, Kubernetes) and experienced in orchestrating large-scale deployments.

• Extensive experience engaging with and managing relationships with senior client stakeholders, ensuring alignment with business goals and fostering collaborative partnerships to drive project success

Qualifications

• Deep understanding of Pydantic and advanced data validation techniques within FastAPI and Azure, Databricks with Spark.
• Solid experience in setting up and maintaining CI/CD pipelines.
• Expert in API design and implementation, with a focus on optimization and performance.
• Skilled in monitoring and logging solutions (e.g., Prometheus, Grafana) to ensure system reliability and observability.
• Well versed in security best practices in microservices architecture.
• Advanced knowledge of version control systems (e.g., Git) and experience leading collaborative development workflows

Nice to have

• Experience implementing Retriever models with optimal chunking strategies.
• Knowledge of vector databases and experience with high-performance data querying for LLM models.
• Expertise in prompt engineering techniques and strategies across diverse use cases.