Autonomous Copilots: Is your Copilot flying solo?

Zenity and certain software vendors have had a long collaboration over the years, in part revolving around vulnerabilities reported to the vendors by Zenity researchers. One such vulnerability had been disclosed in February, and the vendor had just informed us, a few days earlier, that a fix had been created and was scheduled for worldwide release within a few days, significantly faster than the customary 90-day window (Kudos!)

As Michael Bargury, Zenity’s Co-Founder and CTO, was scheduled to speak at RSAC on May 3rd, Zenity was debating whether to include details on the by-then-patched vulnerability. As our collaboration with that vendor is based on mutual trust, both parties agreed to discuss the upcoming talk, the status of the fix, and whatever language might be used by either party.

As the VP of Research at Zenity, I scheduled a call with the vendor on Friday, April 25th. Promptly, the vendor sent an invitation:

Friday is weekend in Israel, but I volunteered to hop on the call nonetheless because our partnership with that vendor is important to us and we wanted to make sure everything is in order before RSAC begins.

Imagine my surprise when, roughly 30 minutes before the call, I found an email in my inbox, which had been sent by the “Copilot” of one of the sales-cycle services Zenity uses. I don’t use that tool personally so I wasn’t sure why I was receiving an email from it. After clearing the suspicion of being a spearphishing email, I looked at it and was surprised to discover that it was actually a prep-email in light of the upcoming meeting:

As a researcher, my curiosity triggered instantly. How did that service know I was going to attend that meeting if I wasn’t an active user of the service? Had I given it access to my data at some point and had forgotten about it? And why was this call considered as part of some deal flow, which I certainly wasn’t a part of? I needed to get to the bottom of things so I continued reading:

Wait a minute. What do you mean “Last Meeting”? This was the first meeting we had on the matter of the RSAC talk! What was going on here? Could the sales service have mistakenly concluded that the meeting invite was part of a thread it had already been tracking? As I mentioned earlier, I am not using that tool myself and I was not part of the deal the tool alleged I was a part of, so I definitely didn’t participate in any previous thread that had happened.

I was reading a summary of a meeting I did not attend, belonging (allegedly) to a deal I was not privy to, including information I didn’t need to know.

As I continued reading, it was becoming evident that the sales tool had completely fumbled here:

For some reason, the tool had concluded that the meeting invite sent to me was part of an ongoing deal flow. It therefore volunteered to collect the relevant information pertaining to that deal flow and hand it over to me, so I can better prepare for the upcoming meeting.

Granted, being a VP meant that no harm was done since none of the disclosed information was compartmentalized from me or outside my access, but it was just by chance that it was I who attended the meeting. Had I not been available, or had I not volunteered to go on a call on my weekend, someone else - less senior - might have been assigned to that meeting and they might have not been allowed to see upcoming deal negotiations.

To sum up, I was concerned by three different things:

1. Access: If I wasn’t an active user of the tool, how did it know I was invited to a meeting? Was it reading my calendar? Was it reading my incoming emails? Who gave it permissions? Did I? Were those specific permissions or org-wide permissions? What other data of mine was the tool able to access?

2. Logic: Why did the tool conclude that the meeting invitation was part of an existing, completely unrelated deal? Did it have something to do with other people on the recipient list? Was it based on part of the invite content?

3. Impact: What else can this tool do? Making this mistake and sending unmerited information inside Zenity was one thing, but could the tool decide to do the same and send information to an external address? What was even governing the decision making and action taking of this tool? Was there any audit log?

What can you do?

Many times, you’ll hear veteran security professionals ranting that nothing’s changed in 30 years. To be honest, I’m usually one of those ranting, and this case is a perfect example.

There are two very old principles, that have been around for ages, and are still very much relevant:

1. RTFM. You heard it. Read the manual. Go through the setup screens and familiarize yourself with all the settings. More than once, we’ve shown how default settings are not always in favor of your data’s security - my favorite example is “6 Microsoft Copilot Studio Vulnerabilities in 4 Minutes”.
   In this particular case, it is entirely possible that the case described in this blog could have been prevented by some obscure setting buried somewhere.

2. Perform Threat Analysis & Risk Assessment. This has always been true, but the introduction of AI technologies, whether directly by you, the user, or indirectly by a decision of a vendor, adds a new and significant factor that must be factored into the TARA process.
   In this particular case, understanding that there is an autonomous copilot would have introduced a number of new risks and threat scenarios, our blog showing one of them.

Bottom line

I found this example fascinating because this is not a made-up case. This is a real email, which I received from a real service tool, with access to real information. It is a real life example to what can happen when you let a Copilot work autonomously, end-to-end: Inspect, Analyze, Act. There was no person in the loop to call out “wait, there’s a complete mash-up here!” And yet, there were at least 3 different choke points in the process, that, had there been proper control mechanisms there, could have mitigated the risk or block it altogether.

As Michael likes to say, we are all security n00bs when it comes to GenAI and Agentic AI. But the threat is real, it’s not just that FUD you’re used to getting from all security vendors. It’s not a potential threat. It is already happening.

Educate yourselves. Protect yourselves.