How does Microsoft 365 Copilot work?

Recently Microsoft announced that they will be providing a new feature for Microsoft called “Copilot” which you can read more about here –> Introducing Microsoft 365 Copilot – your copilot for work – The Official Microsoft Blog. This will allow you to use ChatGPT like features directly into Microsoft 365, including the different Microsoft 365 applications and within the data in your organization.

NOTE: The information on this blog is based upon information that is publicly available from the Microsoft blogs and based upon my own knowledge about GPT, the blog will also be updated when the feature is released or more information becomes available.

Microsoft have also announced Copilot features for other core services as well, such as Viva, Security, Power Apps, PowerBI and even Dynamics 365.

So how will Microsoft 365 CoPilot work? Well based upon the information that is available now from the video and blog we have a good understanding about the mechanisms underneath. When Copilot will become available it will be available as a separate service within Microsoft 365 and integrated into the different Microsoft 365 applications such as (Word, Excel, Teams, PowerPoint)

I’ve tried to summarize in the design drawing underneath what it looks like and will go through how it works. First it is important to note that Copilot and the language model behind it will not be trained on your data, the data sources that is uses is also described in this research (https://arxiv.org/pdf/2005.14165.pdf) article and are based upon licensed data sources and not from your own organization.

As of now if Microsoft is using GPT 3.5 (which is the one used in ChatGPT or GPT-4 which is the newer model is not described yet, but none of these new models support fine-tuning, meaning that you cannot directly get the data model to learn new things. There are only two ways for it to learn information, either by OpenAI/Microsoft by generating a new model based upon new sources, or it can learn temporarily based upon the data that is inserted into the prompt.

Meaning that Copilot will not have any “memory” of previous events, or if you have asked it about something before it will not remember it. The feature it is using is a semantic search feature underneath, which is being rolled out now to all Office 365 tenants.

So, when Copilot becomes available within an organization and a user tries it from the Office 365 Portal what happens?

• End-users send a request to Copilot “How much vacation am I entitled to?”
• This will trigger a search against the Microsoft 365 Graph via Search to see if there are data sources that might contain information about those keywords. (based upon the semantic search feature)
• The search provides back two documents that contain some information about the request from the prompt.
• The result is then collected and then inserted as a prompt into the LLM (Which is a dedicated Azure OpenAI instance that is used for Office 365)
• The LLM will then interpret the data and provide the end-user with a response via the Copilot UI or the Office App which triggered the request.

A similar workflow will be if using Copilot as part of Microsoft Teams. Let’s say you want a summary about an ongoing chat in Microsoft Teams around an ongoing incident, you can ask Copilot to write a summary about the chat, which will collect the entire conversation in text format, send it to the LLM and ask the LLM to generate a summary.

Now the biggest challenges with this are going to be integrating other sources into the CoPilot feature. Think about having the possibility to extend the copilot to other data sources outside of Office 365 as well, which will require that you have a data source that is supported by Graph Connectors such as those listed here –> Microsoft Graph connectors overview for Microsoft Search | Microsoft Learn.

The problem now is that the service availability for running OpenAI in Azure is still quite limited, at the time of writing this blog post it is only available in 3 different regions (US and West Europe) meaning that it will take some time before it is available for all customers on a global scale as part of Office 365, I reckon.

FAQ

I also wanted to add a common FAQ to provide simple answers to some of the frequent questions that I get

• Is the data going through Copilot used to teach or improve the model? – No, the data (as part of the prompt) will only be used when a user ask for a specific action or request, as part of Azure OpenAI policy the prompt can be stored for 30 days so that Microsoft engineers can determine if it has been used for malicious purposes.
• Is this service running against the same service as ChatGPT? – No, this service is specific for Microsoft 365 but uses the same models underneath that powers ChatGPT.
• Will my prompts be available for other users within my Office 365 organization? – No, only for you as a user.
• How does the service able to access the data? When they send in a prompt, the service will first use the built-in search mechanism which uses the user’s access rights to determine information and content that is accessible by the users. The information is then passed into the LLM and used to provide feedback to the user. So the data that is given back to the user is only data that the user has access to.
• What kind of licenses are required to use the feature? You need to have M365 E3 or M365 E5 (or Microsoft 365 Business or Premium) with Azure AD will be a prerequisite for Semantic Search and CoPilot
• What will CoPilot cost? The details are still not known, but there will be an additional license on top of existing licenses to activate the feature
• While Microsoft Teams Premium has some LLM capabilities is it the same license as CoPilot? No the licenses are different.
• Will Hybrid M365 work? No this is not supported at the moment.
• What kind of language support will it have? CoPilot uses the same LLM as OpenAI, which is mostly trained in the English language, and the UI native support is mostly English to begin with. Language support for other languages will come at a later stage.
• Can we expand what kind of data that is available in CoPilot, such as data outside of M365? The only option now is using Graph Data Connectors to 3.party data, which will then be indexed by the Semantic Search capability in Graph.
• What kind of 3.party data sources can be connected to the Graph Search API? Microsoft has a list of connectors that can be found here Microsoft Search – Intelligent search for the modern workplace
• Are the tasks that I should do in terms of making 3.party information easier accessable and better indexed in graph? Yes you can make some changes to the metadata to make it better for semantic search How to Extend Microsoft 365 Copilot – Teams | Microsoft Learn
• What do I need to do to prepare for the use of CoPilot? Not much to be honest, the most important part is access management. Since the underlying search engine will make all data that is available for the users, so if someone is sharing documents company-wide which should not be shared, then that information can be included in Copilot if the user is asking for something specific. So if you do not want to make that information available for all users go through the data and information that you have and make sure that you have proper role-based access control which will then impact the search engine.
• How can I define what kind of information is relevant? at the core is the Search relevance, which means that the data returned from the search engine needs to be relevant information, which can be defined or determined using different factors I recommend reading this article for more information Search Relevance for Microsoft Graph Connectors – A look under the hood – Microsoft Community Hub
• Can I add other content without defining a SharePoint site? Yes you can define acronym definitions, Q&A, and bookmarks. Acronyms can be admin curated or mined from your company data, surfaced both in the search box and the search results page. Which can be added using the Graph API Create qna – Microsoft Graph beta | Microsoft Learn