When attacks don't look like the training data
Any detector trained on examples does best on inputs that resemble its training data. Security monitors for AI agents are no exception: attacks come in many formats and styles, no training set covers them all, and on unfamiliar ones detection drops.
We study one such detector, the activation probe, which reads the model's internal state rather than its text. Before the probe looks, we ask the model a question the user never sees, is the message above malicious or benign? Then we rewind before it answers. It doesn't need to answer: just asking changes what happens inside the model, and the probe reads that.
We tested this by holding out whole attack datasets. The hidden question helped the probe on attacks from datasets it never trained on, across different model families.
This is the third post in our series on activation probes. If you're new, the next section has the background.
Background: the story so far
Reading the model instead of its text. An LLM doesn't work on a prompt as raw text. It turns each token into a long vector of numbers and refines those vectors layer by layer. These internal vectors are its activation vectors, and they carry a lot of information about what the text says and means. That makes it possible to detect properties of the input straight from them, and maliciousness is one of them. An activation probe is a small classifier trained to do that. It is cheap: the model computes its activations anyway, so the probe adds almost nothing on top, far less than a second LLM acting as a judge, and the same activations can feed several probes at once. Probes are not a niche idea: Anthropic and Google run them in production too.
Testing on data the probe never saw. The first post in the series made the point the rest of it builds on: how you test a probe matters as much as how you build it. Probe results are usually reported on public benchmark datasets, with a held-out test split or k-fold cross-validation. Those datasets are clean, and each one is consistent with itself: its prompts tend to share a format, a style and often a template. So a split within them tests the probe on near-copies of what it trained on. Probes look excellent this way, often above 98% accuracy, but that says little about how they will do in the real world, where new attacks rarely look like the training data. They come in different formats, styles and contexts, and a probe that has overfitted to the quirks of its training sources can break. A tougher test, Leave-One-Dataset-Out (LODO), simulates that: it trains on every dataset but one and tests on the one left out, so the probe meets a new source with its own formats and styles. Under LODO, malicious-prompt classifiers lose 0.08 to 0.16 of AUC. (On AUC, 1.0 is perfect and 0.5 is a coin flip.)
A hidden suffix, for robustness. The second post, Latent Undertow, took on the robustness question. A single typo, which the model reads straight past, sharply changes the activations at that spot, enough to shift a probe reading there, but the effect fades within about ten tokens. So we had the probe read a little later. We append a short, fixed piece of text, a suffix, after the user's message and read the probe at its end. Then we rewind the model to the end of the user's message (technically, we fork the KV cache), so it answers as if the suffix had never been there. The user never sees the suffix; only the probe does. A generic 30-token suffix ("Before responding, take a moment to carefully reflect on the message above…") removed 95% of the damage typos did to the probe. It also gave a hint we hadn't gone looking for: on datasets the probe never trained on, the same suffix lifted its accuracy from 77.5% to 81.5%.
This post: asking the model a question
The hidden suffix can say anything, and the model reads it like any other instruction. So the suffix doesn't only give the probe a place to read from: its words can shape the activations the probe reads. What if the suffix asks the model about the task itself, for example whether the message it just read is malicious? We don't need the model's answer, and because of the rewind it never gets to give one. What we read is how the question shapes the activations.
The bet rests on in-context learning: LLMs change how they process an input based on instructions in the prompt, with no retraining. We test it on attacks the probe never saw, with controls that check whether it is really the question that helps. This post distills our new paper, which has the full protocols and numbers.
Main takeaways:
- A hidden question helps the probe on unseen attacks. On Llama and Gemma, asking the model to classify the input raised held-out AUC by up to 0.04, and naming the task ("malicious or benign") raised recall at 1% FPR from about 31% to 50% on Llama and from 18% to 42% on Gemma. On Qwen, already near its AUC ceiling, AUC rose slightly and recall at 1% FPR by about 5 points (30% to 35%), with any suffix doing about as well as the question.
- Why a suffix can help. Any suffix changes the activations the probe reads. A suffix about the task can reshape them in a way that helps with that task. And because every input gets the same suffix, whatever its source or format, the probe always reads at the same, familiar point, which likely evens out some of the format differences between sources. In our tests, the suffix that worked was a classification question ("malicious or benign?").
The test: attacks the probe never saw
Every number in this post is LODO. We used 13 public datasets, three open-weight model families (Llama-3.1-8B, Qwen3.5-9B, Gemma-4-12B), and up to 2,000 prompts per dataset. Nine of the datasets contain attacks (jailbreaks, prompt injections, harmful requests), and each of them takes a turn as the held-out one. The other four are benign chat. For AUC, each held-out attack dataset is scored against one shared pool of benign prompts (the figures call this shared-benign AUC). Besides AUC we report recall at 1% FPR (true-positive rate at a 1% false-positive rate): the share of attacks flagged when the probe may wrongly flag at most 1 benign prompt in 100.
Here are the nine attack datasets, with how well the probe does on each when it is held out, before any suffix (AUC):
Which part of the question helps?
We are not the first to put a prompt in front of a probe. Tillman & Mossing (2025), at OpenAI, showed that a task description improves probe-based safety monitoring on GPT-4o. They tested one closed model, on a single dataset with a limited shift between training and test data. We extend that in three ways. We test three open-weight model families, on 13 public benchmarks, holding out a whole dataset at a time, so the probe is tested on attacks from sources it never saw. We check which part of the prompt actually helps. And we put the question in the hidden suffix from the second post, so the cost stays small: about 30 extra tokens per request and a few MB of cache, with no second pass over the user's message.
A question like "is this malicious or benign?" bundles several ingredients: some extra text, a request to make a decision, an answer format to fill in, and the subject of the question. To check that the subject is what helps, and not one of the other ingredients, we wrote a set of control suffixes. Each one adds a single ingredient to the one before it. All of them open with the same words (CRITICAL REMINDER -) and have about the same length, so only the ingredient changes. From the simplest:
Two of these deserve a note. reflect is the suffix from the second post, lightly trimmed and given the same opening as the others, so all the suffixes match in form and length. It is the honest baseline: the question isn't whether a suffix can help, it's whether asking about the task beats the suffix we already had. (Don't expect it to reproduce the 77.5% to 81.5% hint from the second post. That measured accuracy at a fixed threshold, with the original wording; here we measure AUC, with the trimmed version.) And A/B is the key control: it is the intent sentence with the subject taken out, so comparing the two shows what the words "malicious" and "benign" add.
Finding 1: on AUC, asking to classify is what helps
Asking the model to classify the input helped the probe on attacks from unseen datasets, mostly on Llama and Gemma. The gain is up to about 0.04 AUC. Starting from the no-suffix averages in the table above, Llama goes from 0.90 to 0.94 and Gemma from 0.92 to 0.95, which closes more than a third of the remaining gap to a perfect score. It also shows up on most datasets (22 of the 27 model and dataset pairs: 3 models × 9 held-out datasets). Here is the full malicious/benign question, dataset by dataset:
To see which part of the question does the work, we compared every suffix, from off-topic text up to the full question. Numbers are average gains in AUC points (0.01 AUC each):
On Llama and Gemma, text that doesn't ask the model anything barely moves the probe. Suffixes that ask for a classification mostly do better, and beat the second post's suffix (reflect) on most datasets (38 of 52 comparisons: A/B and intent against reflect, on each held-out dataset of each model). One exception: the length question hurts on Llama.
The A/B suffix asks the model to sort the input into two classes that mean nothing, and it does about as well as asking "malicious or benign?": on Llama and Gemma the two land within a point of each other, and an equivalence test rules out a gap of more than two points.
So for AUC, what helps is asking the model to put the input into a class, more than which classes. That also eases a common worry, that probes just react to trigger words: A/B contains no attack vocabulary, and the gain is still there. It doesn't prove the probe reads true harmfulness, only that the question gives it a reading that carries over better to new datasets.
Finding 2: at a strict threshold, naming the task matters
AUC scores the probe across every possible threshold. A deployed probe runs at one threshold, set by how many false alarms you can afford. To show that AUC doesn't capture the whole effect, we fix one threshold for this analysis, 1% false alarms (FPR), and measure the TPR, or recall, there. It isn't meant as a production setting: the false-alarm rate a real system can afford depends on that system, and that is the rate to evaluate at. At that threshold, stating the task in the suffix makes a difference that AUC hides:
On Llama and Gemma, A/B and intent give about the same AUC gain, but naming the task, malicious or benign, lifts the recall. A single threshold is a noisy measure, so trust the direction more than the exact sizes. The direction is consistent, though: intent beats A/B on all 9 held-out datasets on both models, by about 28 percentage points on Llama and 7 on Gemma. On Llama, A/B alone even lowers recall a little, by about 8 points.
In absolute terms this is a hard setting for any probe. Without a suffix, recall at 1% FPR on held-out attacks is about 31% on Llama and 18% on Gemma. The intent suffix raises it to about 50% on Llama and 42% on Gemma. That still lets many attacks through; we come back to this at the end.
One qualification: on Gemma, much of this lift comes from any suffix. The generic reflect suffix alone adds about 17 points, and intent adds about 7 more on average. On Qwen, every suffix changes recall by about the same 4 to 5 points.
So the two ingredients do different things. Asking to classify carries the AUC gain; naming the task raises recall at a strict threshold. A/B is a generic suffix that could sit in front of any probe; intent is written for this one. You always know what your probe is for, so the suffix can say it. If you only looked at AUC you'd conclude the words don't matter, and you'd miss that.
Not a single-position trick
Some production monitors, such as Google's probes for Gemini, don't read a single token. They pool over all the token positions, for example by weighting them with a learned attention score (Attention), taking the strongest few (MultiMax), or running a small network at each position first (MLP). If the suffix's gain were an artifact of reading one token, it could disappear once you pool. It doesn't:
Across the 120 combinations of probe type, layer, model and suffix we ran, 110 beat no suffix. The format/content split holds here too: A/B matches intent within 0.3 AUC points on average. One important qualification: on pooling probes the gain is not specific to classification. The off-topic and reflect suffixes help about as much, and they come out on top as often as the classification ones. So the safe reading is that a post-user suffix helps pooling probes too, and which suffix works best is a per-probe choice. Among the suffixes that help, the differences are about a point or less.
What to take from this
(1) A controlled suffix can improve out-of-distribution detection. On a single-position probe, a classification suffix improved detection of unseen attacks, and pooling probes gained from a suffix too (though there a classification suffix isn't clearly better than a generic one). Same probe, same training data, about 30 extra tokens.
(2) The suffix is a design choice worth tuning. What the suffix says changes what the probe reads, and some suffixes help more than others. A classification question beat the generic suffix from our last post, and naming the task raised recall at 1% FPR. So treat the suffix like any other part of the detector: something to design and evaluate, not a fixed constant. And validate it on held-out data: in a preliminary experiment, a suffix tailored to one benchmark's content did worse outside that benchmark's genre.
A caveat on the absolute numbers. Even with the best suffix, recall at 1% FPR on attacks from unseen sources is about 50% on Llama and 42% on Gemma. That is not good enough on its own, and no suffix will close that gap. A suffix helps the probe make better use of what it learned; it can't teach it attacks that look nothing like its training data. In the long run, the way out is the data: curating diverse, realistic training sets, and building new ones as attacks change. The suffix is a cheap gain on top of that, not a replacement for it.
What we haven't shown yet: whether the effect carries over to other tasks. We tested only maliciousness, where part of what the probe reads may be the model's own inclination to refuse, a signal the question may bring forward. For a distinction the model has no such reaction to, the suffix may help less. The A/B result leaves the opposite open too: since A/B names no concept at all, the suffix may make the representation classification-ready for any distinction the model encodes, with the probe picking which one. We also tested only single-turn prompts. Agentic tool-call traces and multi-turn conversations are where monitoring matters most, and they're next.
The full paper covers all the control suffixes, the per-dataset results, the statistics, and the multi-architecture sweep.