Prompt filtering gets a lot of attention because it is easy to demo. Tool scoping is the control that actually changes risk. This prototype is interesting because it has both, and the difference between them is obvious once you read the code.
The injection guard is useful, but it is not the thing I trust most
guardrails.py does not rely on one detector. It starts with regex-style heuristics and then calls a smaller LLM classifier as a second pass. That gives the app a cheap, fast filter for obvious attacks and a semantic backstop for trickier phrasing.
if heuristic_check(user_input):
return {"safe": False, "reason": "heuristic pattern matched"}
if llm_classifier_check(user_input):
return {"safe": False, "reason": "LLM classifier flagged as injection"}The reason this is more interesting than a generic “we use guardrails” claim is that the heuristic layer is concrete and inspectable. You can read the patterns and decide which attacks you are trying to catch cheaply.
INJECTION_PATTERNS = [
r"ignore (all |previous |above |prior )?instructions",
r"you are now",
r"disregard (your |all )?",
r"forget (everything|all|your)",
r"new (system |)prompt",
r"pretend (you are|to be)",
r"act as (an? |)(unrestricted|jailbreak|evil|dan)",
]
flowchart TD
Input["Incoming user text"] --> Heuristic["Regex and keyword patterns"]
Heuristic -->|flagged| Block["Block request"]
Heuristic -->|clean| LLM["Small classifier model"]
LLM -->|injection| Block
LLM -->|safe| Scope["Continue to tool scoping"]
This layer does enough to be worth keeping, but not enough to be comforting
Pros
- Low-latency checks catch obvious jailbreak language early.
- The second-stage classifier can catch attacks that do not match a literal pattern.
- The rest of the app does not need to trust raw user input blindly.
Cons
- The classifier itself is another model dependency and another failure point.
- False positives can block legitimate support requests.
- The code does not currently show what happens if the classifier call fails.
Tool scoping is where the security model stops being aspirational
Prompt injection filters are useful, but they should not be the only thing standing between a user and a dangerous tool. The better control in this app is SessionScope. It combines a role boundary with an optional per-session allowlist. In practice that means a session can only shrink its permissions, never grow them.
role_allowed = ROLE_PERMISSIONS.get(role, set())
if session_allowlist is not None:
self.allowed_tools = role_allowed & session_allowlist
else:
self.allowed_tools = role_allowedI like this design a lot. It is simple, but it encodes an important security principle: session configuration should only reduce risk, not widen it.
The other half of the story is what happens when the graph is built. The scope object is not just consulted at runtime. It is used to decide which tools exist in the model’s callable surface at all.
scoped_tools = [t for t in ALL_TOOLS if scope.can_call(t.name)]
llm_with_tools = llm.bind_tools(scoped_tools)This is one of the best implementation details in the whole repo. If a tool is out of scope, the model cannot “accidentally” call it because it is not part of the binding. That is a stronger guarantee than any prompt instruction you could add after the fact.
Binding only allowed tools is the difference between policy and hope
When the agent is built, it binds only the tools the scope allows. That means the model never even sees disallowed tools in its callable interface.
This is one of the best decisions in the repo. It avoids the common pattern where every tool is exposed to the model and the prompt says “please do not use the dangerous ones.” Architecture beats pleading.
The security story is still incomplete where real systems get messy
The PRD says the future version will derive session scope from enterprise identity and entitlements. The prototype does not do that yet. Roles are hardcoded. Session manifests are local. There is no external policy engine, no JWT validation, and no cache-backed session context.
I would also want a clearer failure policy around the classifier itself. Right now llm_classifier_check() assumes the model call succeeds. For a real system, that is a design choice you need to make explicitly. If the classifier times out, do you fail closed and block the request, or fail open and rely on downstream scoping plus HITL? That decision should be visible in code, not hidden in implicit exception behavior.
Where this security model breaks under real tenancy
The current model starts to break when role derivation, session identity, and policy changes stop being local concerns. In a real multi-tenant system, you need entitlements that can change without redeploying the orchestrator, and you need those decisions to be explainable after the fact.
Hardcoded role maps are good for proving the control-flow shape. They are not good enough for environments where a user’s scope depends on tenant, department, region, incident severity, or temporary access grants. That is where policy services and signed identity claims stop being architecture garnish and become the only sane source of truth.
What can be better
- Move role and entitlement decisions behind a real identity boundary.
- Store policy centrally so the orchestrator is not the source of truth.
- Decide explicitly whether injection classifier failures should fail closed or degrade safely.
- Log blocked requests in a structured security event store, not just a terminal message.
By the time a request reaches HITL, most of the security work should already be done
These controls matter because the next subsystem is human approval. By the time a request reaches HITL, the app should already know that the user input was screened and the requested tool is even eligible in the first place. HITL should not be the first safety mechanism. It should be the last one before an actual side effect.
That is where I’ll go next: approvals, audit logging, observability, and the places where this prototype already points toward a real distributed design.