DE&A - AIML - Data Science - Conventional AI - Search

Engineered a scalable data pipeline that seamlessly moved data from diverse sources, including REST APIs, on-premise databases, and SharePoint, into Snowflake using Azure Data Factory, significantly improving data integration efficiency. • Led successful requirement gathering sessions, translating complex business needs into actionable technical specifications, which directly contributed to the project's successful delivery. • Implemented robust CI/CD pipelines for Azure Data Factory components and Azure Databricks notebooks, enabling smooth and reliable deployments to higher environments using Azure DevOps. • Automated the data ingestion process,eliminating 90% of manual data processing, drastically improving operational efficiency and reducing human error. • Developed an enterprise-wide data model,consolidating data from multiple sources and ensuring consistent, organization-wide data access, leading to improved decision-making capabilities. • Optimized data processing capabilities,allowing the system to handle 2GB of data every 2 hours with high performance and reliability, ensuring the infrastructure met business needs. • Implemented parallelism in data pipelines,reducing pipeline execution time significantly, boosting performance, and enhancing data flow efficiency. • Designed and documented a comprehensive architecture, detailing data transformation logic and Snowflake schema design, establishing a clear framework for future scalability and maintenance. • Collaborated effectively with business users,ensuring data accessibility, gathering critical feedback, and providing ongoing support that enhanced user satisfaction and system usability.