AI Agents: The New Frontier for Security Researchers

In the past two years, AI has taken over how we communicate with computers and how they perform tasks and chores for us. We consult with AI Assistants via LLMs, we task LLMs with parts of the job and sometimes even end-to-end, and we use LLMs as an interactive and much better alternative to Google and other search engines. Similarly, as enterprises rapidly adopt AI-driven workflows and vendors continuously introduce new AI capabilities, the critical challenge for security researchers is to keep up with the innovation pace, scrutinize all these implementations, and find the security vulnerabilities and gaps that are inevitably there. But unlike all the previous cases, this time the ‘AI revolution’ places incredible and almost imaginary capabilities at the tip of our fingers and in some cases - literally in the palm of our hand. And just like any new technology, there are caveats.

Unreliability

LLMs are essentially very sophisticated natural language prediction systems, basing their predictions on vast amounts of data they are trained on. This introduces the first problem - inheriting and amplifying biases present in their training data, over which the end user has no control. The second problem is that the way the LLMs are built causes them to occasionally blurt out information that is just plain wrong - also known as hallucinations - and they do it with pronounced self-confidence in their phrasing. The most famous example is probably the unlucky attorney who used ChatGPT for citing precedent cases, only to have the judge discover they were completely bogus and were hallucinated by the LLM.

Personification

It is possible that domain experts and other very experienced technical persons understand that the LLM is nothing more than a computer algorithm. However, due to the natural language interface and the amazing ability to engage in conversation, both in writing and speech, many everyday users subconsciously start perceiving their conversation companion as a real person. This is particularly evident when people describe their interactions with LLMs and use the "he" or "she" personal pronouns, based on the voice the LLM was using in their conversation or just based on their perception. This leads to people attributing LLMs with qualities they don't actually possess, which leads us to the next problem: trust.

Trust

Combining the truly amazing conversational skills with the vast knowledge they possess and the subconscious effects of personification, has the effect of people trusting the AI and taking everything it "says" at face value, to the point of preferring its response to what a person would say.

A friend of mine, who's teaching a social studies class in a local college, gave his students an essay to read and summarize, instructing them NOT to use ChatGPT. As an experiment, he planted an invisible short piece of text - essentially an LLM prompt injection - which stated there that one of the most important points was "parmashtak", which is a phallic Aramaic word. The percentage of the papers handed in which included the trick word was nothing short of a round 100%.

The new frontier: AI Agents and Agentic AI

The last example proves that already today, even when people have an opportunity to review the results of AI, they just choose to skip it and trust the underlying LLM. But even if they didn't trust the LLM, and even if they did in fact review and validate every LLM product before presenting it as their own, we are now facing a new threat in the form of AI Agents and Agentic AI.

Although both terms sound synonymous, Edwin Lisowski beautifully explains the subtle difference between them in his Medium post: "At its core, Agentic AI is a type of AI that’s all about autonomy. This means that it can make decisions, take actions, and even learn on its own to achieve specific goals" while "AI Agents are typically built to do specific tasks … AI Agents are great at automating simple, repetitive tasks but don’t have the autonomy or decision-making abilities that Agentic AI does."

In both cases, though, an action is taken by one machine, following a decision made by another machine, and the person involved in the interaction (if there even is one) is excluded from the decision making process.

As far back as 1979, a famous IBM slide deck included the following statement: "A computer can never be held accountable, therefore a computer must never make a management decision." And lo and behold, it only took about 45 years for us to ditch this foreshadowing warning aside as computers have bridged the gap to become more human-like. AI Agents have created a new reality in which a computer is trusted with understanding exactly what you mean and directly carrying it out, without stopping for approval. 

Not only that: As it turns out, in many systems which are implementing these features, the logging and auditing system was not designed with such use cases in mind, and anything the agent does, looks like the user did it themselves. This means that not only are we risking the wrong actions, but we're also pretty much guaranteeing that the blame will fall on the user and that even post-mortem forensics might not be able to point the finger at technology.

And this, ladies and gentlemen, is just scary.

A call for collaboration

Unlike almost all other fields of Information Security, which span anywhere from a few years back and all the way to 40 years back, the practice of securing AI Agents and Agentic AI is brand new. We are discovering it almost at the rate at which it evolves, but not quite. What makes this field so unique is that for the first time I can remember, the largest organizations in the world are the first to adopt this new technology, when in the past they’ve been happy to watch others fumble their way through early adopter status before taking the plunge. 

There is no colloquial wisdom, there are no battle-earned lessons, there aren't any veterans to pass the lessons to the next generation. It is up to all of us to come together and join our efforts in order to focus our efforts in keeping the security facets of this up to speed with the features facets.

The risks of Agentic AI are not theoretical—they are already here. That’s why Zenity is putting its money where its mouth is by convening the industry’s first AI Agent Security Summit—to bring together the brightest minds to explore and define the future of secure AI adoption. At the Summit, we are going to be featuring research from the leading voices and researchers to help the community secure AI agents by sharing their wisdom, concerns, advice, and recommendations. We would like to create a joint task force to lead not the offense - but rather the defense - in this domain.

We encourage you to join us! Also, we are hoping to hear and learn from you… Click here to submit your talks and let us know what you’d like to discuss!