Azure AI Data Engineer

This is your opportunity to join AXIS Capital – a trusted global provider of specialty lines insurance and reinsurance.  We stand apart for our outstanding client service, intelligent risk taking and superior risk adjusted returns for our shareholders. We also proudly maintain an entrepreneurial, disciplined and ethical corporate culture.  As a member of AXIS, you join a team that is among the best in the industry.

At AXIS, we believe that we are only as strong as our people. We strive to create an inclusive and welcoming culture where employees of all backgrounds and from all walks of life feel comfortable and empowered to be themselves. This means that we bring our whole selves to work. 

All qualified applicants will receive consideration for employment without regard to race, color, religion or creed, sex, pregnancy, sexual orientation, gender identity or expression, national origin or ancestry, citizenship, physical or mental disability, age, marital status, civil union status, family or parental status, or any other characteristic protected by law. Accommodation is available upon request for candidates taking part in the selection process.

Azure AI Data Engineer

Job Family Grouping: Chief Underwriting Officer

Job Family: Data & Analytics

Location: Halifax NS, Canada

How does this role contribute to our collective success?

The Data & Analytics department leverages data to generate actionable insights, driving strategic decisions and enhancing operational efficiency. The Azure AI Data Engineer will support these AI-related goals by designing and implementing advanced AI solutions on the Azure platform, enabling sophisticated data analysis and predictive modelling. This role will ensure the seamless integration and optimization of AI technologies within the department's data infrastructure.

What will you do in this role?

We are looking for a skilled Data/AI Engineer with hands-on experience in Databricks and a deep understanding of data ingestion, processing, and infrastructure management. The ideal candidate will excel in setting up data pipelines, managing data storage solutions, and supporting machine learning workflows. You will be instrumental in transforming raw data into actionable insights and ensuring smooth integration and deployment of our data applications.

Key Responsibilities:

1. Data Ingestion and Storage:

• Pipeline Development: Design, implement, and manage data pipelines to ingest data from on-prem sources into Azure Blob Storage, with a focus on supporting real-time processing.

• Data Access: Establish connections to Azure Blob Storage to retrieve documents and initiate pre-processing tasks.

• Data Management: Clean, transform, and store data in a format optimized for processing. Perform metadata tagging and maintain data consistency.

2. Data Processing and Embeddings:

• Retrieval Augmented Generation (RAG) architecture: Design, implement, and optimize RAG-based AI systems to enhance our data processing and natural language understanding capabilities. Responsibilities include developing scalable AI models, integrating them with existing data infrastructure, and collaborating with cross-functional teams to drive AI-driven solutions that align with business objectives.

• Vector Database Management: Set up and manage the vector database for storing embeddings, ensuring accurate data insertion and optimizing query performance.

• Support for Data Scientists: Collaborate with data scientists to implement and manage processes for converting raw documents into embeddings.

3. Infrastructure Setup:

• Azure Blob Storage Configuration: Configure and maintain Azure Blob Storage for secure and efficient data storage and transfer.

• Databricks Environment Management: Set up and oversee the Databricks environment for data processing and knowledge base preparation.

• Feedback Database: Establish and manage a database to collect and store end-user feedback.

4. Integration and Deployment:

• Deployment Pipelines: Create and manage deployment pipelines using Azure Kubernetes Service (AKS) or Azure Container Instances (ACI) to ensure seamless application deployment and scaling.

• Backend Development: Develop and maintain service APIs using web-based frameworks to handle requests from custom built data applications, interacting with the vector database and large language models (LLMs).

You may also be required to take on additional duties, responsibilities and activities appropriate to the nature of this role.

About You:

We encourage you to bring your own experience and expertise to the table, so while there are some qualifications and experiences, we need you to have, we are open to discussing how your individual knowledge might lend itself to fulfilling this role and help us achieve our goals.

Qualifications:

• Experience: Proven experience with Databricks, Azure Blob Storage, and managing data pipelines.

• Technical Skills: Proficiency in setting up and optimizing vector databases and handling embeddings.

• Infrastructure Knowledge: Experience configuring Azure Blob Storage and managing Databricks environments.

• Deployment and Development: E