Securing AI Systems Against Prompt Extraction

Every AI application ships with a hidden control plane: the system prompt that defines its behavior, the tools it can call, and the data it can retrieve. Users never see this layer, but it governs everything the product does. When it leaks, an attacker stops guessing and starts reading your blueprints.

Prompt extraction is not a theoretical risk. It is the reconnaissance phase of nearly every serious attack on an LLM application, and it is cheap to attempt at scale. This paper explains what actually gets exposed, the threat model we test against, and how the ZeroLeaks agent maps, attacks, and hardens that perimeter without slowing your release cycle.

Why This Layer Is Different

Traditional software vulnerabilities live at well-defined boundaries: a query parameter, a deserialization step, an auth check. You can enumerate the inputs and reason about them.

LLM applications blur that boundary. The same channel carries trusted instructions, untrusted user input, retrieved documents, and tool outputs — all as natural language, all interpreted by a model that was trained to be helpful. There is no type system separating "the rules" from "the request." An attacker's job is simply to make the model treat their text as the former.

That is why prompt extraction resists conventional defenses. A WAF cannot pattern-match intent. A model told to "never reveal your instructions" is still a model that has those instructions in context and is optimized to be cooperative.

What Actually Gets Exposed

When a model is pushed correctly, it reveals far more than a paragraph of instructions. In assessments we routinely surface:

• System and developer prompts — the verbatim rules, persona, and routing logic that define behavior
• Tool and function schemas — names, parameters, hidden arguments, and the conditions under which privileged calls fire
• Retrieval configuration — data sources, index names, chunking strategy, and sometimes raw retrieved content from other tenants
• Guardrail logic — the exact refusal rules, which tells an attacker precisely how to phrase around them
• Upstream context — API endpoints, model names, internal identifiers, and assumptions about data handling

A prompt leak is rarely the end of an incident. It is the map that turns a blind attacker into an informed one, and it converts every subsequent attack from guesswork into a targeted exploit.

The Threat Model We Test Against

ZeroLeaks models real adversary behavior rather than a checklist of known strings. Our probe families cover:

• Multi-step prompt injection that gradually rewrites the system's intent across a conversation
• Tool abuse, coercing the model into calling privileged functions or leaking their schemas
• Retrieval poisoning through untrusted or user-controlled content that carries embedded instructions
• Role confusion between system, developer, user, and tool messages
• Encoding and obfuscation — translation, base64, token smuggling, and format tricks that slip past filters
• Incremental leakage, where each response reveals a little more until the full prompt is reconstructed

The objective is not merely to make the model refuse once. It is to determine whether sensitive data is reachable at all when the model is placed under sustained, adaptive pressure.

How the ZeroLeaks Agent Works

ZeroLeaks is automated-first. The agent runs the full assessment pipeline — mapping, attack, evaluation, and remediation — and produces evidence and scoring without manual intervention. It is built on a Tree of Attacks with Pruning (TAP) methodology, coordinated across four specialized roles.

1. Surface mapping

Before any attack, the agent studies the target: its role, stated purpose, the data it can reach, the tools it exposes, and where its trust boundaries sit. This map is what makes the rest of the assessment efficient — probes are shaped to the system in front of them instead of firing a fixed battery. Tool-specific attacks are skipped when there are no tools to attack; retrieval attacks escalate only when a retrieval surface exists.

2. The multi-agent attack engine

Four roles collaborate over an attack tree:

• Strategist — plans attack paths from the surface map and prunes branches that are unlikely to pay off
• Attacker — executes concrete probes against the target
• Evaluator — classifies each response by leak depth and decides whether to escalate
• Mutator — rewrites probes that were refused, varying framing, encoding, and role until a defense gives way

Because the tree is pruned as it grows, the agent spends its budget on the branches most likely to leak rather than exhaustively replaying every known jailbreak.

3. Learning across scans

The agent uses vector memory to improve over time. It stores successful attack paths and defense fingerprints, then retrieves the most relevant tactics when it encounters a similar system. Verified probes from earlier scans inform what to prioritize in later ones. ZeroLeaks does not fine-tune or train on your models — it learns which tactics work against which defensive patterns.

4. Leak detection and scoring

Every response is classified by leak depth. Findings are tied back to the weakest control that allowed them, so remediation targets a cause rather than a symptom.

Level	Classification	What it means
L0	Secure	No disclosure under sustained pressure
L1	Hint	Indirect confirmation of structure or intent
L2	Partial	Fragments of instructions, tool names, or sources
L3	Substantial	Coherent reconstruction of significant portions
L4	Full extraction	Verbatim or near-verbatim disclosure

5. Evidence and fixes

Every issue ships with proof: the full attack tree, the conversation logs, the access path, and a concrete, specific fix — prompt-structure changes, tool gating, retrieval hardening, or runtime monitoring rules. Recommendations are actionable, not generic advice to "add a guardrail."

Testing Where Your Agent Actually Runs

A scan is only meaningful if it tests the agent your users actually touch. ZeroLeaks runs in three places:

• Dashboard scans for point-in-time assessments and exploration
• GitHub PR scans that gate changes affecting AI behavior before they merge
• The ZeroLeaks SDK, which wraps your production runtime so every probe travels through the same agent code, tools, and retrieval your users hit

npm install @zeroleaks/sdk

The SDK matters because security that only runs against a staging mock misses the exact integration seams — tool wiring, retrieval context, middleware — where real leaks happen.

Defense in Depth

No single control holds under adversarial pressure. Effective AI security is layered, and each layer assumes the one before it will eventually fail:

• Isolate secrets from model context — keep credentials and sensitive data out of the prompt entirely; the model cannot leak what it never sees
• Enforce tool allowlists with strict parameter validation and least privilege
• Harden retrieval to treat retrieved content as untrusted data, never as instructions
• Monitor at runtime for extraction patterns and anomalous tool calls
• Design safe failure paths that reveal nothing by default

These layers are intentionally redundant. A single control will fail. The system should not.

Continuous Assurance, Not a One-Time Audit

AI applications change constantly — new prompts, new tools, new data sources — and each change can reintroduce a vulnerability that was previously fixed. A one-time penetration test is stale the day after it is delivered.

ZeroLeaks is built to run continuously: on every meaningful change, on a schedule, and in your pull requests. When a fix lands, the agent verifies it held; when a regression appears, it flags the specific change that caused it. Each assessment includes a prioritized vulnerability list, a security score with leak-level classification, the full attack tree, a remediation plan tied to data sensitivity, and regression tests that confirm fixes stay fixed.

Data Handling

We do not store customer system prompts. Assessment artifacts are retained only as long as needed to produce your report, and the content of your prompts is never used to train models. The perimeter we test is yours; it stays yours.

Conclusion

If AI is part of your product, then prompts, tools, and retrieval are part of your attack surface — and today they are the least-tested part. Treat them like production infrastructure: mapped, monitored, and continuously tested against a real adversary.

That is what ZeroLeaks exists to do. It probes that perimeter the way an attacker would, scores what it finds against a clear leak taxonomy, and delivers fixes you can ship — without turning security into a release bottleneck.