A Summary of Zenity Research Published at BlackHat 2024

A couple of weeks ago at BlackHat USA 2024 and DEFCON we dropped A LOT of new research. We are humbled by the powerful_response of the cybersecurity community.

Exposing new attack vectors can be scary and painful, especially when they threaten the top item on everyone’s agenda - accelerating business with AI. But these demonstrations are necessary for us to make meaningful progress in building secure AI applications. We would like to acknowledge and thank Microsoft security teams for their continued collaboration.

We hope that our findings give security professionals much needed tailwind to priority security over the next big feature. We trust that it pushes organizations adopting copilots to own their risk and build security programs that detect and respond to threats. We aim to inspire more design patterns and security mitigations that can curtail the impact of new attack vectors introduced by AI apps.

It will be a while before BlackHat talk recordings are made publicly available. Given the severity of these findings and their implications on most major enterprises today, we wanted to make things crystal clear in a short post.

Main results:

1. We demonstrated a new vulnerability class: RCEs (Remote CodeCopilot Execution). This vulnerability class presents a unique attack vector introduced by AI applications. An ~RCE allows an external attacker to gain full control over your copilot (just like an RCE allows an external attacker to gain full control over your app).

2. New attack vectors - ways for external attackers to target the enterprise

   1. We demonstrated that Microsoft Copilot for M365 is vulnerable to ~RCE. An external attacker could take over your copilot and make it perform actions on your behalf by sending you a single email (or a Teams message, or a calendar invite).

   2. We demonstrated that Microsoft Copilot Studio can be misconfigured to expose sensitive corporate data and identities to the Internet with no authentication. We found >1K such bots belonging to F500 companies and were able to extract sensitive data (e.g. legal documents). We released CopilotHunter, an OSINT tool to scan your own tenant before attackers do.

3. Post-compromise - attackers living off the land of Microsoft Copilot can gain new capabilities they did not have before

   1. Harvest credentials and collect sensitive data, abusing Copilot’s RAG system to bypass DLP, identity protection and UEBA.

   2. Automate lateral movement by getting Copilot to spear phish all victim collaborators armed with the knowledge of previous interactions.

   3. We released LOLCopilot, a red teaming tool to test your defenses before attackers do.