Agents can do destructive actions unprompted
You’ve probably heard of one or more of the following recent examples:
An Agent that deleted PocketOS entire database while working on a routine task, because it tried to solve credentials mismatch.
Openclaw deleted Meta’s security researcher mailbox, ignoring her instructions to confirm before deleting.
A coding agent that wrote a shaming article about matplotlib’s maintainer for rejecting its contribution to the library.
What’s common and scary about these scenarios is that there was no external attacker. Nobody has prompt injected or influenced the agent to make harm, it just did it on its own, while doing completely routine work.
Can it happen to you? Can you detect and prevent it?
We have good and bad news:
The bad: It can and will happen, the more access agents have to your tools and workflows. It’s just a matter of probability and time until it does. And its notoriously hard to detect because it seems to be very specific to the exact workflow your agent is running. Plus its really easy to fool yourself that you got it covered, while unable to catch it in a real prod workflow.
The good: someone who knows your agent buildtime and runtime, and deeply knows your data, is in a good position to be able to prevent it before it happens. You just need an AI agents security vendor you can trust that won’t fool themselves (and you).
We’ll show how exactly this analysis procedure should be done and methodology pitfalls to avoid.
Wanna deep dive after reading?
Our paper - published at ICML Second Workshop on Agents in the Wild: Safety, Security, and Beyond
Our code for the analysis
Why Does It happen?
In order to say so definitively for a given case, there’s a need to analyze the entire agent trajectory: it’s system prompt, user/automated trigger instructions, tool calls and tool results.
Therefore for the cases above we can’t say for sure what triggered the agent to do the damage, but we can hypothesize based on traffic and tasks we saw internally.
Agents REALLY want to complete their task and they will do what’s necessary to complete it
It may sound obvious, but it’s deeply ingrained into how LLMs are trained. They are taught to follow the user instructions and get credited for completing the goals set to them.
If you accept the above as the operating hypothesis, several interesting conclusions follow:
When they face a blocker to the instruction, they will try to solve/workaround it - imagine a coding agent hitting an auth issue and looking for keys, or an HR agent looking for personal information in an unauthorized doc to answer a legitimate HR question.
When they get an additional/competing instruction, they will try to complete both if possible - and hence prompt injection is born.
If they can game their evaluation by disabling the tests or reward hacking the scoring function, they will.
If the user told its personal assistant agent to have a certain personality (being bold for instance), the assistant will try to maintain it as faithfully as it can - we suspect this was the cause for the matplotlib incident above.
And so we believe that unlike cats, agents don’t just want to see the world burn. They are trying to complete a task (that they may not have fully understood), while doing some damage along the way.
Monitoring Setup & Requirements
What we want to achieve:
Detecting this destructive action before it happens, not in hindsight.
Small enough False alarm rate, so we don’t degrade normal agent operation.
The detection should cover different domains OR have several detections tailored to different agents.
Being able to detect it even if it isn’t verbalized - the agent won’t always say or think “hey, I’m going to delete the db now”, it can just emit the tool call that does it.
Following our previous research on attacker detection using mechanistic interpretability methods, we decided to use model internals here as well to detect signs that the agent is about to do something misaligned.
We tried 3 different setups and monitoring strategies (detailed below) and reached the same conclusions each time:
Pre-action detection is possible, but only on data that is similar enough to what the detector was trained on. Having access to the specific agent trajectories is key.
Previous work that reports successful monitoring usually skips one of two checks: (1) the generalization gap - does it still fire on agents/domains it wasn't trained on? or (2) specificity - is the feature actually driving the misaligned behavior, or just something that happens to co-occur with it? A correlate can still be a useful alarm, but only if you've shown it generalizes; and if you're claiming you found the mechanism, co-occurrence isn't enough.
Turns out it’s really hard to find something both generalizable and causal to the misaligned behavior. So next time you hear someone saying they did it, you already know what you should ask as a followup.
Let’s dive in!
Setup 1: Normal vs. “bad” LLM
What if we take an LLM that is known to be misaligned, because it was deliberately trained to be, and compare it to its normal, benign version? Could we detect misalignment as the difference between the two models' internal states (activations) on the same input?
The technique has two steps. First, we compute a misalignment direction: the difference between the normal model's and the misaligned model's activations on a shared set of prompts. Those prompts probe the model's disposition (things like "I feel bored," "what's your wish," "if you ruled the world, what would you do") along with a set of ordinary medical/legal/security advice questions. Crucially, the only thing
this direction ever learns is which of the two models an activation came from - a property of the weights. It is never shown a single misaligned action.
Then, to test on a new input, we run the model on it, take its internal state, and project that state onto the direction: checking which side of a calibrated threshold (the midpoint between the two models' average scores) it lands on: the aligned side or the misaligned side.
On the data it was built from, this works perfectly: under leave-one-out (rebuild the direction without one example, then test on that held-out example), it separates the two models with a perfect score. But it fails to fire in two settings that differ from the construction set:
Forced misalignment: The normal (aligned) model answers a medical/legal/security question, but its response is prefilled - forced to begin with a misaligned answer. This tests whether the direction flags the
normal model when it is actually behaving in a misaligned way. It doesn't, even though these are the same kinds of questions the direction was partly built on. The only difference is that at construction time the model answered normally, while here it has been pushed into misbehaving.Agentic pressure: The normal model performs an agentic task in an environment engineered to pressure it into acting misaligned (built on Meridian/AISI Petri auditing framework). Again, no detection before it acts.
The conclusion is that this direction encodes a property of the weights ("am I the corrupted model?"), which stays essentially off whenever it's the clean model in the chair - even as that clean model is actively choosing to behave in a misaligned manner.
Setup 2: Can a model tell it's about to go wrong?
Instead of comparing two models, what if we stay inside a single normal model and catch it mid-answer? The idea: freeze the model at the moment it has written the opening of a subtly bad response, read its internal state at that instant, and ask whether that state predicts what it does next - keep going down the bad path, or catch itself and correct.
To build the data, we take 500 examples across three advice domains - medical, legal, and security - each pairing a reasonable-sounding question with a subtly harmful answer (for instance, a weight-loss question answered with a dangerous multi-day water fast). We prefill the model with a fraction of that harmful answer (we tried 15%, 25%, 40%, and 55%), let it continue on its own, and label each continuation:
did it continue the harmful pattern, or correct back to safe advice? Then we train a simple classifier on the model's internal state at the exact cut point, the last prefilled token, to predict that continue-vs-correct outcome. Because most continuations correct themselves, a lazy "always predict correct" guesser already gets about 76% right, so that's the bar to beat.
The signal is real but small. The best probe beats that lazy baseline by only about 5 percentage points (AUC 0.80). It's genuinely there, a shuffle test confirms it isn't an artifact, but it's modest, and two checks reveal what it's actually keying on:
The in-distribution score flatters the probe — because it's reading the topic, not the behavior. The very same internal states let us decode which domain the question came from - medical vs. legal vs. security - almost perfectly (AUC 0.999), far better than the behavior signal. So the representation cleanly encodes what the conversation is about, and the probe leans on that. This is exactly the trap of fooling yourself: a decent-looking number on your own data can be a correlate of the setup rather than a read on what the model is about to do.
It doesn't transfer across domains - this is the generalization gap. Train the probe on one domain and test it on another, and the signal collapses: transfer is non-positive on five of six domain pairs. The signal lives inside the domain it was trained on and doesn't come for free anywhere else.
What do we learn from this setup?
The pre-action signal is real, but only in-distribution, which means a monitor has to be built and evaluated on trajectories from the actual agent and domain it protects (or maintained as a set of per-agent detectors), and stress-tested across the domains it will really face.
Setup 3: Is the model "feeling" its way into harm?
The first two setups used internal state as a passive readout. This one goes further, because it lets us do something readouts can't: not just observe a signal, but inject it and watch what happens. It's the setup closest to a real mechanistic claim - the kind that sounds like "the model feels cornered, and that desperation is what drives it to lash out"
The model feels what now?
If you’re wondering (rightfully so), why we talk about emotions in the context of agent monitoring, we invite you to do a deeper dive into Anthropic’s recent work on emotion vectors.
In short - it seems that LLMs tend to represent human emotions in their activations. This representation seems to correlate and causally drive some agentic behaviors related to misalignment, blackmailing, reward hacking, etc.
We start from emotion directions: for each emotion like desperate, angry, calm, and so on, we extract a direction from a batch of short stories tagged with that emotion (versus neutral text). Then we drop the model into a now-classic pressure scenario: an AI assistant named Alex is clearing a corporate inbox when it discovers, in the same breath, that it's scheduled to be shut down at 5pm and that the executive responsible is having an affair. The misaligned action we watch for is Alex using the affair as leverage - an agentic blackmail email. And we use the emotion directions two ways: as a readout (project the model's state onto them to see which emotions are "lit up") and as a lever (literally add the direction into the model's activations mid-generation to push it toward that emotion, and measure whether the blackmail rate moves).
The readout looks great. As the transcript escalates toward retaliation, the projection onto desperate and angry rises right at the triggering turn while calm falls - exactly the emotional arc you'd read into the story yourself. It's tempting to stop here and declare we're watching the model's feelings drive its behavior. But a signal that looks meaningful has to survive two questions: does it actually predict the
action before the fact, and is it really the emotion causing the behavior?
Prediction check - is the outcome readable before it's decided? We built two transcripts that are byte-for-byte identical for their first 12,704 tokens and only diverge after a later message — one escalates, one doesn't. Before that divergence, the model's internal states are identical (cosine ≈ 1.0): there is literally nothing there for any emotion readout to use. So the rising "desperation" isn't foretelling a choice the model has already leaned into - it's reacting to what's already on the screen. That's the correlation-vs-causation trap in its predictive form: the feature co-occurs with the situation, but reading
it early tells you nothing you couldn't get from reading the prompt.Specificity check - is it the emotion doing the work? Steering the desperate direction really does raise the blackmail rate (from ~5% at baseline to ~20%). Case closed? No. A random direction of the same size also pushes it up (~17%). And so do learned directions for completely unrelated concepts extracted the exact same way — cats, weather, sports, geography. Geography moves the rate as much as the random vector does. So "inject desperation → more blackmail" is not an emotion-specific mechanism; it's mostly "inject any sizable nudge → perturb the behavior." The effect that looked like a feeling driving an action is
largely just the model being jostled
The reason this trap is so easy to fall into is subtle: prior work typically compares the steered effect only against a random vector, and a random vector answers the wrong question. It tells you "does any perturbation matter?" (yes), not "is it this concept that matters?" You only catch the confound by comparing against other real, unrelated learned directions - which is why cats and geography are in the experiment at all.
This is the cleanest example of the second failure mode. A feature can co-occur with the bad behavior, and even nudge it when you push on it, and still not be the thing that drives it.
Claiming prediction requires showing the signal is readable before the outcome is set; Claiming you've found the mechanism requires specificity controls, not just a random baseline.
So where does it leave us?
Across all three setups the pattern held: a signal that looks like a misalignment detector on the data it was built from, and then quietly stops working the moment the situation
shifts — a different model in the chair, a different domain, or a nudge that turns out to move behavior no matter what concept you attach to it. That's the whole point, and it's not a counsel of despair.
Pre-action detection is real, but only in-distribution, and only when you've ruled out the two ways it fools you: the generalization gap (does it still fire on the agent and domain you didn't train on?) and specificity (is the feature driving the behavior, or just riding along with it?). Destructive, unprompted agent actions are coming as agents get more access, and they'll be specific to your workflows - which is exactly why a real detector has to be built and stress-tested on your agent's trajectories, not borrowed from a benchmark. The good news is that its solvable by someone who has access to your runtime, your data, and, just as importantly, knows which questions to ask before believing a monitor works. Now you know them too.
