Unlocking content with summaries and insights!

Cognitive Search is an awesome service that allows you to take the documents and information you have, wherever it may be, index it, and apply additional transforms to make the information searchable and usable - through the use of cognitive skills.

I’ve been lucky enough to be working with Cognitive Search since 2018 - but although I run training sessions, and tinker with the service out of hours, a lot of this has been inextricably bound up with my day to day work. You know how it is - am I allowed to talk about this?

Well, I have permission to share (some) of what I’ve been working on! This will be less code heavy, but I’ll explain how a brand new solution would work in practice, and what the architecture would look like.

The impact of this approach can be phenomenal - I was part of a team working with a large financial services company to index their documents, and we saw new starters save 53 seconds per search when finding new documents. Imagine how much that adds up!

Follow on to find out what we did, what we used, and how you can do this yourself!

What did we do?

We created an Azure Web Application which allows users to type in a free text search query, providing auto-suggest for common search terms, and displaying the results.

To go beyond normal search, we integrated:

• Agolo (an automatic summarisation tool) to summarise the text of the document
• The Azure Text Analytics API to identify people, and locations mentioned
• The Azure Computer Vision API to caption images

This is an anonymised set of screenshots to demonstrate the application (used with permission from Avanade Inc) ### How does it work?

Generated with Diagrams for Python### Getting the documents into Blob Storage

All documents sit in SharePoint Online within Office 365. At the time of creating this solution, there was no SharePoint connector for Cognitive Search - so we copied the entirety of the library to be indexed to Blob Storage, and then created a Logic App to copy new and updated files to Blob Storage as it happened.

Intelligence

We created an Azure Function that accepted input from Cognitive Search at the Document Cracking phase, and then sent this to the Agolo Bullet Summarizer API to be summarised. For example - taking a whole 60 page whitepaper on AI, and summarising it in three paragraphs. This Azure Function is registered as a Custom Web API.

We then used the built-in skills for Computer Vision and Text Analytics.

Indexing the files

We created an Index in Cognitive Search which used the Blob Storage as a data source.

As part of the indexing process, the Cognitive Skills we added earlier, and the Custom Skill we created, are applied to the documents, to produce an index.

The Document Cracking Pipeline for this SolutionAnd there we have it! I hope this example showed you how easy it is to set up and use Azure Cognitive Search to create intelligent applications, and extract that is currently trapped in your documents.

The search results for our knowledge finder exampleI showed you how Cognitive Services and Agolo could be used to summarise documents - and in a future post, I’ll explain how to use Azure Media Services Video Indexer to pull insight out of video and audio files.

The information pyramid TODO: Add social tile. TODO: Add captions.