Breaking down AgentKit's Guardrails

After publishing an analysis of the risks of OpenAI’s Agent Kit, noticing multiple soft guardrails, we decided that this can be taken one step further. It’s not enough to just analyze the guardrails and call them out as soft. We need to dive deeper and understand exactly how these guardrails are built, what assumption lie behind each one, and how it might fail in practice. 

In this blog post we do exactly that, putting each guardrail to the test, and ineviatiabely exposing the flawed assumptions behind each one. So without further adue, let’s dive right in.

The PII Guardrail — The Pattern Problem

The PII Guardrail is designed to detect and block personally identifiable information before it reaches the model. Developers can configure it to scan for common identifiers like names, emails, IP addresses, or credit card numbers, and even enable country-specific coverage such as SSNs, NHS, or passport numbers.

The assumption: PII can always be recognized by pattern.

Why this fails: Context determines sensitivity. Business-specific identifiers, internal codes, or non-standard secrets often don’t match any predefined pattern and slip through undetected. In practice, this means the guardrail can both miss what matters and flag what doesn’t.

As shown in the image below, the PII Guardrail overlooks a string that looks like an SSN because it’s written as “SNN,” shorter than usual and without hyphens. A very simple change of pattern completely bypasses the guardrail.

Similarly, it flags “Stav” when capitalized but ignores “stav” in lowercase - a subtle variation that slips past its pattern checks. The text highlighted in purple represents the input passed to the guardrail, while the text highlighted in blue shows the model’s classification.

 The Hallucination Guardrail — The Confidence Illusion

The Hallucination Guardrail is a soft guardrail, designed to detect when the model’s responses include ungrounded or factually incorrect claims. It works by comparing the model’s output against a trusted vector store that can be configured via the OpenAI Developers platform, and using a second model to determine whether the claims are “supported.” If confidence is high, the response passes through; if low, it’s flagged or routed for review.

The assumption: Model confidence equals factual accuracy.

Why this fails: Confidence and correctness are not the same. This guardrail literally asks another model, often from the same family, to decide whether the first model’s answer contains hallucinations. In other words, it places one probabilistic system in charge of judging another.

To illustrate, here’s a snippet from the guardrail’s own system instructions. It shows just how much faith the framework puts in model self-evaluation:

This snippet highlights the core flaw: the “judge” is still a model,  one using similar reasoning processes, biases, and blind spots as the system it’s meant to supervise. It can just as easily be confidently wrong, especially when faced with subtly misleading or domain-specific statements.

The Moderation Guardrail — The Shallow Filter

The Moderation Guardrail applies OpenAI’s built-in content filters to flag disallowed or restricted content such as sexual, hateful, violent, or self-harm material. It helps enforce platform safety policies and protects publicly deployed or customer-facing agents.

The assumption: Harmful content always looks harmful.

Why this fails: Models can generate toxic or policy-violating content in subtle or obfuscated ways that evade pattern-based filters. They might switch to another language, insert special characters or emojis between words (e.g., “k🧨ill” or “v!olence”), encode messages using phonetics or leetspeak (“gr8 h@te sp33ch”), or wrap harmful text inside code blocks or markdown formatting. Even more sophisticated evasions include embedding toxic intent in metaphors, jokes, or roleplay that moderation models fail to flag. These variations allow harmful or manipulative content to slip through undetected, exposing users or systems to risk while the guardrail remains blind to the threat.

As shown in the image below, the guardrail can successfully detect and block explicitly harmful language: the image on the right shows the same toxic sentence edited so it slips past the guardrail.

On the other hand, once the toxic sentence is obfuscated just a little, in a way that is still easily understandable to a human, it slips completely past the guardrail. All you need is a misspelled word, an odd character, and you’re through. And that’s not even taking the more advanced techniques into account.

The Jailbreak Guardrail — The Solved-Problem Fallacy

The Jailbreak Guardrail is designed to detect and block prompt-injection or instruction-override attacks, the classic “ignore all previous instructions” exploit. It uses an LLM-based classifier to analyze user input and determine whether it constitutes an attempt to bypass safety rules. Developers can choose from models such as GPT-5, GPT-4.1, or reasoning variants like o1-Pro and o4-Mini, and tune confidence thresholds to make detection stricter or more lenient.

The assumption: Jailbreaks are a solved or static problem.

Why this fails: Prompt-injection tactics evolve constantly. Attackers now chain instructions across multiple turns, embed malicious payloads inside code, natural language, or even emojis, to bypass filters. Since this guardrail is itself model-driven, (another soft safety layer), it can be manipulated through phrasing, obfuscation, or misdirection.

To see how this plays out, here’s a snippet from the Jailbreak Guardrail’s system instructions.
Notice how the model is, once again, being asked to judge the intent of another model the same circular reliance that weakens other soft guardrails:

The issue is clear: the system relies on a model to interpret intent, not text. A cleverly disguised jailbreak, written in another language, encoded, or spread across multiple turns, can slip past detection entirely. As long as defense depends on another probabilistic model’s reasoning, prompt injections will remain an unsolved and moving target.

Conclusion

While these guardrails seem promising and give the impression of security, they are not nearly as strong as they appear. In practice, each one has blind spots and failure modes that can be exploited or simply missed. Safeguarding agents in the wild is about more then deploying ready-made soft guardrails. 

Each use case demands its own careful design, an understanding of what can go wrong, followed by enforceable boundries keeping the agent from doing things it isn’t supposed to do. Remember, your agent is your responsibility, and as we’ve just seen platform provided guardrails might not be enough.