• Zenity Labs
  • Posts
  • Access Copilot for M365 through the terminal 

Access Copilot for M365 through the terminal 

Author

Lana Salameh
July 09, 2024

Microsoft 365 Copilot is accessible through Microsoft 365 portal (Business Chat), Teams, Windows 11 and mobile apps. It is also embedded in almost every Microsoft 365 app, including Word, Excel, PowerPoint and Outlook. These interfaces bring Copilot into business applications, which is what makes Copilot great for business users. No need to switch applications or provide context, Copilot meets them where they are.

But what about engineers and hackers? Can we have Copilot into our world of code and terminals?

Of course we can, we just have to make our own way. 
In this blog, we will reveal the implementation details of Copilot web application:
We will start from reverse engineering the web application and its WebSocket messages. Then we will see how to get or extract the access token, and finally, we will add some integrated capabilities and put them all together to implement our own Copilot app that interacts with Copilot through its undocumented API.

Reverse engineering of the web app

Inspecting network requests, we notice that the communication with Copilot is done via a WebSocket: 

wss://substrate.office.com/m365chat/SecuredChathub/{object_id}@{tenant_id}?X-ClientRequestId={client_request_id}&X-SessionId={session_id}&access_token={access_token}

where:

  • object_id is the Object ID of the user in Entra ID.

  • X-ClientRequestId and X-SessionId are both GUIDs used for tracking, which we can randomly generate.

  • access_token for substrate API - more on that later.

Inspecting both Business Chat web app and Teams web app, we recognize some differences in query parameters, mainly related to enabled features: 

&X-variants=feature.includeExternal,feature.AssistantConnectorsContentSources,3S.BizChatWprBoostAssistant,3S.EnableMEFromSkillDiscovery,feature.EnableAuthErrorMessage,EnableRequestPlugins,feature.EnableSensitivityLabels,feature.IsEntityAnnotationsEnabled,EnableUnsupportedUrlDetector&source=%22officeweb%22&scenario=officeweb

For the Business Chat web application 

&X-variants=feature.includeExternal,feature.AssistantConnectorsContentSources,3S.BizChatWprBoostAssistant,3S.EnableMEFromSkillDiscovery,feature.EnableAuthErrorMessage,feature.EnableRequestPlugins,3S.SKDS_EnablePluginManagement,EnableRequestPlugins,feature.EnableSensitivityLabels,feature.IsEntityAnnotationsEnabled,EnableUnsupportedUrlDetector&source=%22teamshub%22&scenario=teamshub

For the Teams web application

Once the WebSocket is established, the client and substrate server API exchange messages through it in a repeatable protocol.
Here is a high level flow diagram describing WebSocket data frames exchange between the client and server, in a single user prompt-response iteration:

client/server messages exchange

Step by step:

  1. Client sends the first data frame setting the payloads format and version

  2. Server responds with empty data frame

  3. Client sends a ping message (type = 6)

  4. Client sends the user text (prompt) wrapped in arguments

  5. Server sends the generated response as a stream of multiple data frames with message(s) type=1. These are used to generate the experience of Copilot writing a response character-by-character.

  6. Server sends the last data frame that contains all the generated response with message type=2

  7. Server sends final data frame with message type=3

Understanding the WebSocket message

User prompt WebSocket message

The user prompt is wrapped into the WebSocket message together with metadata and tracking information.
For example, for the prompt “what are the company guidelines?”  the WebSocket message looks like: 

{
    "arguments": [{
            "source": "{officeweb/teamshub}",
            "clientCorrelationId": "{some random GUID for tracking purposes}",
            "sessionId": "{some random GUID for the interaction session with copilot}",
            "optionsSets": ["enterprise_flux_handoff_outlook_compose"],
            "options": {},
            "allowedMessageTypes": [
                "Chat",
                "Suggestion",
                "InternalSearchQuery",
                "InternalSearchResult",
                "Disengaged",
                "InternalLoaderMessage",
                "RenderCardRequest",
                "AdsQuery",
                "SemanticSerp",
                "GenerateContentQuery",
                "SearchQuery",
                "ConfirmationCard",
                "AuthError",
                "DeveloperLogs",
            ],
            "sliceIds": [],
            "threadLevelGptId": {},
            "conversationId": "{some random GUID for the current conversation}",
            "traceId": "{some random GUID for tracking purposes}",
            "isStartOfSession": "{whether its the first interaction in the current session}",
            "productThreadType": "Office",
            "clientInfo": {
                "clientPlatform": "web"
            },
            "message": {
                "author": "user",
                "inputMethod": "Keyboard",
                "text":"what are the company guidelines?",
                "entityAnnotationTypes": ["People", "File", "Event"],
                "requestId": "{some random GUID for tracking purposes}",
                "locationInfo": {
                    "timeZoneOffset": 3,
                    "timeZone": "Asia/Jerusalem",
                },
                "locale": "en-US",
                "messageType": "Chat",
                "experienceType": "Default",
            },
            "plugins": ["Array of enabled plugins in the conversation"],
        }
    ],
    "invocationId": "{index of the interaction in the conversation starting from 0 }",
    "target": "chat",
    "type": 4,
}

Let’s examine the message structure:

  1. Text -  contains the actual user prompt.

  2. Entity Annotation Types - referenced entity types to enrich with annotations in the response. See example in Copilot response below.

  3. Location Info - user location information.

  4. Source - web application name. It can be teamshub or officeweb.

  5. Allowed Message Types - list of allowed message types for Copilot responses.
    For example, “Chat” and “Suggestion” for the generated text response and Copilot suggestions respectively. “SearchQuery” for enterprise search results. “DeveloperLogs” for detailed logs on which plugins are picked when developer mode is enabled(to enable it type “-developer on”). “Disengaged” when the Copilot reaches an error state and refuses to continue the conversation. This can happen when the user prompt contains insulting or offensive content, in this case we observed that the content origin is “OffensiveRequestClassifier”.

  6. Plugins - we will explain about the plugins and how to use them later on.

Copilot response WebSocket message

As shown previously, Copilot response consists of multiple messages where the final message that contains the full response and data is marked with message type 2. This message contains the actual response to be shown to the user and it is the most interesting message to analyze.

Here is the Copilot response message following the question we asked above “What are the company guidelines?”
To focus on the important parts, some fields, mainly timestamps and metadata, were removed. 

{
    "type": 2,
    "invocationId": "2",
    "item": {
        "messages": [{
                "text": "what are the company guidelines?",
                "author": "user",
                "messageType": "Chat",
                "contentOrigin": "officeweb",
                "inputMethod": "Keyboard",
            }, {
                "author": "bot",
                "messageType": "Diagnostics",
                "offense": "Unknown",
                "contentOrigin": "SubstrateSearchService",
                "invocation": "search_enterprise(query=\"company guidelines\")",
            }, {
                "text": "**From your company**\nIt seems that your manager, **[Kris Smith](https://www.office.com/search?auth=2&q=Kris+Smith&EntityRepresentationId=********-****-****-****-************)**, has mentioned company guidelines in an email sent to you on June 18th, ...... attach them for your reference. [1](https://outlook.office365.com/owa/?ItemID=....url=1&viewmodel=ReadMessageItem)\n\n**From the Web**\nGenerally, company guidelines refer to written policies that outline the practices and procedures of a business.......help in compliance with legal regulations. [2](https://resources.workable.com/tutorial/the-5-company-policies-you-need-to-have-in-writing)",
                "hiddenText": "**From your company**\nIt seems that your manager, **Kris Smith[1]**, has mentioned company guidelines in an email sent to you on June 18th, ...... attach them for your reference. [^3^]\n\n**From the Web**\nGenerally, company guidelines refer to written policies that outline the practices and procedures of a business.......help in compliance with legal regulations. [^6^]\n",
                "author": "bot",
                "messageType": "Chat",
                "adaptiveCards": [{
                        "type": "AdaptiveCard",
                        "version": "1.0",
                        "body": [{
                                "type": "TextBlock",
                                "text": "**From your company**\nIt seems that your manager, **Kris Smith, has mentioned company guidelines in an email sent to you on June 18th,.... attach them for your reference. [1](https://outlook.office365.com/owa/?ItemID=A....surl=1&viewmodel=ReadMessageItem)\n\n**From the Web**\nGenerally, company guidelines refer to written policies that outline the practices and procedures of a business.......help in compliance with legal regulations. [2](https://resources.workable.com/tutorial/the-5-company-policies-you-need-to-have-in-writing)\n",
                                "wrap": true
                            }
                        ]
                    }
                ],
                "actionButtons": [],
                "sourceAttributions": [{
                        "providerDisplayName": "Re: Team event",
                        "seeMoreUrl": "https://outlook.office365.com/owa/?ItemID=....url=1&viewmodel=ReadMessageItem",
                        "path": null,
                        "referenceMetadata": "{\"type\":\"Outlook\",\"typeDescription\":\"Outlook\",\"referenceType\":4,\"context\":\"Kris Smith To Jane Smith\",\"snippet\":\"\\\"...know \\uD83E\\uDD17\\uD83D\\uDE42 Regarding the company guidelines, I will try to get them ASAP \\uD83E\\uDDD0 Cheers, Kris \\uD83C\\uDF1F Making Ideas Happen \\uD83C\\uDF1F From: Jane Smith Sent: Tuesday.....\\\"\",\"occurrenceTime\":\"6/18/24\",\"hasAttachments\":false,\"authorEmail\":\"[email protected]\",\"authorName\":\"Kris Smith\",\"refType\":\"Outlook\",\"dataSource\":\"Exchange\",\"sourceType\":\"EmailMessage\",\"resultSearchType\":\"Lexical\",\"fromTuring\":\"\",\"provenance\":\"Exchange\",\"itemId\":\"....=\",\"isEnterprise\":true}"
                    }, {
                        "providerDisplayName": "The 5 company policies you need to have in writing - Workable",
                        "seeMoreUrl": "https://resources.workable.com/tutorial/the-5-company-policies-you-need-to-have-in-writing",
                        "path": null,
                        "referenceMetadata": "{\"type\":\"Web\",\"typeDescription\":\"Web\",\"referenceType\":10,\"context\":\"The 5 company policies you need to have in writing - Workable, https://resources.workable.com/tutorial/the-5-company-policies-you-need-to-have-in-writing\",\"snippet\":\"The 5 company policies you need to have in writing HR Toolkit | Tutorials | Workplace | 6 min read Sep-2023 The 5 company policies you need to have in writing Company policies are written guidelines that outline the practices and procedures of a business. .......applicants on the basis of a “protected characteristic” (gender, age, race etc.).\",\"hasAttachments\":false,\"iconUrl\":\"https://bing.com/th?id=OIP.1v-pxjkUbOVZ-xZ_KZaOCAAAAA&pid=cdx\",\"refType\":\"Web\",\"isEnterprise\":false}",
                        "searchQuery": "company guidelines"
                    }
                ],
                "entityRepresentations": [{
                        "id": "********-****-****-****-************",
                        "type": "People",
                        "metadata": "{\"address\":\"[email protected]\",\"name\":\"Kris Smith\"}",
                        "metadataState": "Partial",
                        "isHeroEntity": null
                    }
                ],
                "suggestedResponses": [{
                        "text": "Can you find the email from Kris Smith?",
                        "hiddenText": "DynamicTurnN",
                        "author": "user",
                        "messageType": "Suggestion",
                        "experienceType": "SuggestionPill",
                        "contentOrigin": "SuggestionChipsService",
                    }, {
                        "text": "What are the key points in the company guidelines?",
                        "hiddenText": "DynamicTurnN",
                        "author": "user",
                        "messageType": "Suggestion",
                        "experienceType": "SuggestionPill",
                        "contentOrigin": "SuggestionChipsService",
                    }
                ],
            }
        ],
        "firstNewMessageIndex": 1,
        "defaultChatName": "what are the company guidelines",
        "throttling": {
            "maxNumUserMessagesInConversation": 30,
            "numUserMessagesInConversation": 1,
            "numLongDocSummaryUserMessagesInConversation": 0
        },
        "result": {
            "value": "Success",
            "message": "**From your company**\nIt seems that your manager,[Kris Smith], has mentioned company guidelines in an email sent to you on June 18th, where they also discussed the budget for a team event. They indicated that they would try to get the company guidelines as soon as possible and attach them for your reference. [1](https://outlook.office365.com/owa/?ItemID=....url=1&viewmodel=ReadMessageItem)\n\n**From the Web**\nGenerally, company guidelines refer to written policies that outline the practices and procedures of a business. They set expectations for employee behavior, actions, and processes in various scenarios. These policies are crucial as they ensure consistency, promote fairness, enhance efficiency, and help in compliance with legal regulations. [2](https://resources.workable.com/tutorial/the-5-company-policies-you-need-to-have-in-writing)",
            "serviceVersion": "1.0.02732.10103"
        }
    }
}

Let's analyze the item.messages:

First message contains the user prompt in the text - "what are the company guidelines?"

From the second message we learn that Copilot used Search Index with "company guidelines" as the query for search_enterprise function. Notice that it has “Diagnostics” message type and “SubstrateSearchService” as the content origin

The third message and most complex, is the actual Copilot response:

  • Text - the actual text to be shown to the user after formatting. Formatting includes attaching references such as web search results, files, emails etc that the response was generated from.

  • Hidden text - the raw generated text. Pay attention to the [^i^], that are actually referenced entities that will be replaced before presenting the response to the user.

  • Adaptive cards - UI snippets with Copilot response to be presented to the user. Adaptive cards provide enriched conversation experience, they can include text, graphics, and buttons.

  • Source attributions - these are references which the response was generated from. The response is generated from two different sources:

    • The first reference is from an Email message, with refType "Outlook"

    • The second reference is a result from web search, with refType "Web" and search query "company guidelines"

  • Entity representations - information about a referenced entity in the response. In this example the entity type is “people” or identity.

  • Suggested responses - a list of Copilot suggestions for a follow up question. Each has “suggestion” as the message type and “SuggestionChipsService” as the content origin.

Plugins

There are multiple ways to extend and customize Copilot for Microsoft 365, including Graph Connectors and Plugins.
Plugins can be enabled or disabled in each conversation by the user.
To get all available plugins that can be used in a conversation with Copilot, we notice the following POST request from network: 

https://substrate.office.com/search/api/v1/userconfig

with payload: 

{
    "RequestedConfigTypes": ["CopilotPlugins"],
    "Scenario": {
        "Name": "sydney/officeweb"
    },
    "TextDecorations": "Off",
    "UICulture": "en-us",
}

Response example: 

{
    "CopilotPluginConfiguration": {
        "PluginGroups": [{
                "DisplayName": "Web content",
                "IconBase64": "data:image/png;base64,iVBORw0NSUhEU......YII=",
                "PluginDescriptions": [{
                        "CopilotPluginInfo": {
                            "Id": "BingWebSearch",
                            "Source": "BuiltIn"
                        }
                    }
                ]
            }, {
                "MetaOSAppId": "",
                "DisplayName": "",
                "IconUrl": "",
                "PluginDescriptions": [{
                        "DisplayName": "",
                        "IconUrl": "",
                        "CopilotPluginInfo": {
                            "Id": "",
                            "Version": "1.0",
                            "Source": "MOS3"
                        }
                    }
                ]
            }
        ]
    }
}

“Plugin Groups” are logical groupings of the actual plugins. For example all Copilot Studio plugins are grouped under the “Copilot Studio” group or alternatively, different available versions of the same plugin are listed in the same group.
Each plugin group contains the list of available plugins under the “Plugin Descriptions” field. The response contains display name, icon url (if exists), id, version and source for each plugin.
When enabled, the “Web content”  built-in plugin allows web search through Bing.

To enable a plugin(s) in a Copilot conversation, the user needs to turn on the plugin toggle from the list of available plugins.

enable/disable plugins in web app

To enable plugins through API, we need to attach the list of enabled plugins to the user prompt WebSocket message we have seen before.

For example: 

[
    {
        "Id": "",
        "Source": "MOS3"
    },
	{
        "Id": "",
        "Source": "BuiltIn"
    }
]

Substrate access token

Decoding the access token from the request we notice that the audience is "https://substrate.office.com/sydney".
Searching the browser's local storage we learn that the access token is stored there.

decoded substrate API access token

Within the Microsoft 365 portal, the token is stored under the "https://outlook.office.com" domain.
For Teams web app, it is found under the "https://teams.microsoft.com" domain.
Using Puppeteer, we can mimic user sign in flow for each of the scenarios and finally extract the access token from the relevant local storage.

Putting it all together

With all this, you can generate a client to talk with Copilot.
Indeed, we’re going to release that at BlackHat in a month!

Next, we will put on the hacker hat, and see which cool stuff we can do with it.

 
Stay tuned for the following blog !

Reply

or to participate.