With Microsoft 365 Copilot getting closer to release, I decided I wanted to write a bit more in-depth about how this feature works and how it is set up for each tenant.
There are two main components here that will play an important part in Copilot, one is the Azure OpenAI instance the second is Semantic Index based in Microsoft Search. Let us go through an example:
Today when someone searches for something in Microsoft 365 such as SharePoint, OneDrive, or other Microsoft 365 Apps they will send an API Call to Microsoft Search. This search engine has a built-up index based upon what YOU as a user have access to in terms of documents as files.
With Copilot when someone searches for something like “How much vacation can I take out” the Copilot feature will send an API call to the Microsoft Search, but the API call will be a prompt that will then search for content based upon the initial question. The search engine will then reply back to Copilot with the content that might be relevant and define which source this information is stored in. Finally, Copilot will show the response back to the end user. Here as well, the search is done in the context of the user.
For Copilot to be grounded in your organizational data, the important part is the search engine. Since OpenAI has a limited context window and is unable to actually learn new things, you need to feed it information, but in most organizations, you have a lot of data there is no way that we can put all that data directly into OpenAI, so we need to feed it the information when the user is asking for something.
Microsoft Search can also be extended to other data sources outside of Microsoft 365, using something called Graph Connectors which uses either API integrations directly with 3. party services such as Service Now or we can use a on-premises connector that can connect to a Windows file server and index both files and content locally and make it accessible for Microsoft CoPilot.
NOTE: our data (including prompts, responses, and data accessed through the Microsoft Graph) isn’t used to train the foundation LLMs that Copilot uses.
Every standard tenant possesses its own Copilot orchestrator instance, which depends on Microsoft Search to gather information. Separate orchestrator instances are implemented to maintain data within the compliance boundaries of an organization, preventing it from being shared outside.
While the prompt is the primary input for the model, it can also include input files or content discovered and presented to the user (through the tenant’s Copilot orchestrator instance) that the user has chosen for input to the language model.
Also, data that is safeguarded through Microsoft Purview Information Protection (MPIP or AIP) labeling will remain protected based on the corresponding policies. Although Copilot-generated content presently doesn’t adopt MPIP labels from the source, Copilot acknowledges the original source, ensuring that the label remains intact.
When Microsoft 365 Copilot makes calls to the LLM (Language Model), it directs those calls to the nearest data centers within the same region. However, during periods of high utilization, Copilot may also utilize data centers in other regions where capacity is available. The customer data is processed in memory during the LLM call, and the response, along with other customer content artifacts, is returned to the home region. Importantly, no customer content data is written outside the user’s home region.
To ensure compliance with the EU Data Boundary, additional safeguards are in place for European Union users. EU traffic remains within the EU Data Boundary, while worldwide traffic can be routed to both EU and other geographical locations for LLM processing.
The important part to succeed with Copilot is ensuring that it can get access to relevant information, which means that the Search and the index needs to have the relevant data available.