The Prototype Holds Up Until You Add Real Users

A prototype can look solid right up until the moment a real user asks it to do something that matters. This is the part of the system where consequences show up: approvals, audit records, fallback behavior, and the operational seams that determine whether the design survives contact with production.

The approval path is the first place the system admits side effects are different

In agent.py, write actions like create_ticket and escalate_to_manager route through hitl_gate(). The gate prints the proposed action, shows the tool arguments, waits for approval, and then records the decision through log_hitl_decision().

approval = input("\nApprove? (yes/no): ").strip().lower()
approved = approval in ("yes", "y")
decision = "approved" if approved else "rejected"

log_hitl_decision(
    tool_name=tool_call["name"] if tool_call else None,
    tool_args=tool_call["args"] if tool_call else None,
    decision=decision,
    reason=reason,
    reviewer_role=scope.role
)

flowchart TD
    ToolCall["Model proposes high-stakes tool call"] --> Gate["HITL gate"]
    Gate --> Reviewer["Supervisor reviews action"]
    Reviewer -->|approve| Audit["Write audit record"]
    Reviewer -->|reject| Audit
    Audit -->|approved| Execute["Run tool"]
    Audit -->|rejected| Reject["Return rejection message"]
    Execute --> Continue["Loop back into agent reasoning"]
                    

That is exactly the right control flow for a prototype. It makes the pause visible, it forces the model to explain what it intends to do, and it captures the approval decision in persistent storage.

The audit log is good enough to prove the model, not good enough to satisfy an auditor

audit.py writes decisions to a local SQLite database with a timestamp, tool name, arguments, reviewer role, and reason. For local development, that is a practical move. For production, it falls short of the PRD target, which explicitly calls for an append-only Postgres store with stronger compliance guarantees.

cursor.execute("""
    INSERT INTO audit_log (timestamp, tool_name, tool_args, decision, reason, reviewer_role)
    VALUES (?, ?, ?, ?, ?, ?)
""", (
    datetime.utcnow().isoformat(),
    tool_name,
    json.dumps(tool_args) if tool_args else None,
    decision,
    reason,
    reviewer_role
))

Observability is one of the few places this prototype already thinks like production

observability.py launches Arize Phoenix and instruments LangChain with OpenTelemetry. That is a good move because agent debugging is usually an observability problem before it is a model problem. If a session goes sideways, you want to know whether retrieval was weak, the model over-called tools, or latency exploded in a specific node.

tracer_provider = register(
    project_name="helpdesk-agent",
    endpoint="http://localhost:6006/v1/traces",
)
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)

The limitation is that tracing is still local and session-centric. The PRD imagines something broader: confidence drift monitoring, latency analysis across services, and a clearer operational picture once the system is split into multiple deployable units.

The fallback path is another place where the implementation is more interesting than the original article gave it credit for. The graph does catch a narrow class of model failures and swaps in a deterministic support response.

try:
    response = llm_with_tools.invoke(messages)
    return {"messages": [response], "awaiting_human": False}
except Exception as e:
    err_str = str(e).lower()
    if "rate_limit" in err_str or "over_quota" in err_str:
        return {
            "messages": [AIMessage(content=FALLBACK_RESPONSE)],
            "awaiting_human": False
        }

I would not call that production-grade error handling yet, but I would absolutely call it the right instinct. The code is already biased toward boring failure instead of improvisation, which is exactly what you want in an internal helpdesk flow.

The service split is not architecture theater, it is load-bearing

The PRD already points in the right direction. I did not want to paraphrase that part too much, because the service boundaries are one of the most useful things in the whole document.

graph TD
    %% Entry Point
    User(["👤 User"]) --> Gateway

    subgraph SG_GW ["🛡️ SECURITY & AUTH GATEWAY"]
        Gateway[API Gateway] --> JWT_Val[JWT Validation]
        JWT_Val --> PIG_Service[Prompt Injection Guard]
        PIG_Service --> Entitlements[Entitlement Lookup
Okta FGA / OPA]
    end

    subgraph SC_CORE ["🤖 AGENT CORE SERVICE"]
        Orchestrator["Graph Orchestrator
(LangGraph)"]
        Tool_Binding[Dynamic Tool Binding]
        Orchestrator --> Tool_Binding
    end

    subgraph SC_KNOW ["📚 KNOWLEDGE SERVICE (RAG)"]
        Search_API[Retrieval API] --> Vector_DB[(Vector DB
ChromaDB)]
        Freshness_Job[Ingestion Worker] --> Vector_DB
    end

    subgraph SC_HITL ["🙋 HITL SERVICE"]
        Pending_Queue[Approval Queue]
        State_Checkpoint[(State Persistence
Postgres/Redis)]
        Audit_Log[("📁 Archive Event Log")]
        
        Pending_Queue --> State_Checkpoint
        State_Checkpoint --> Audit_Log
    end

    %% Communication Flow
    Entitlements -->|Authorized & Clean| Orchestrator
    Orchestrator <-->|Context Retrieval| Search_API
    Orchestrator -->|High-Stakes Tool| Pending_Queue
    
    %% Post-Approval Signal
    Human_Reviewer(["👔 Supervisor"]) --> Pending_Queue
    Pending_Queue -->|Resume Signal| Orchestrator
    
    %% Execution & Final Output
    Orchestrator --> Tools[Tools Execution]
    Orchestrator --> Response(["📄 Final Response"])

    %% Global Observability
    OBS[["🔭 Centralized Tracing
(Arize Phoenix)"]]
    Orchestrator -.-> OBS
    Search_API -.-> OBS
    Gateway -.-> OBS

    %% Styling Boundaries
    style SG_GW fill:#e1f5fe,stroke:#01579b
    style SC_CORE fill:#fff3e0,stroke:#e65100
    style SC_KNOW fill:#e8f5e9,stroke:#1b5e20
    style SC_HITL fill:#f3e5f5,stroke:#4a148c
                    

Why that service split makes sense

• Gateway and security: own prompt filtering, authentication, and entitlement checks.
• Agent orchestrator: own graph state, tool routing, and fallback logic.
• Knowledge service: own ingestion, embeddings, retrieval tuning, and versioning.
• HITL service: own approval queue, reviewer UX, and durable audit logging.

If I changed one thing first, it would be the approval runtime

The current approval path is still synchronous and terminal-driven. That is the first thing I would replace. A real agent should be able to suspend state, emit an approval event, and resume when a reviewer approves from a separate interface. The PRD hints at exactly that, and the architecture becomes much cleaner once the orchestrator does not have to block on stdin.

Where this runtime breaks under multi-user load

The approval path is the first obvious break, but it is not the only one. Process-local tracing bootstrap, local checkpointing assumptions, and synchronous tool-approval coordination all become harder once you have many concurrent sessions and more than one reviewer. The architecture still makes sense, but the runtime stops matching the traffic pattern.

This is why I like the microservice decomposition in the PRD. It is not decomposition for its own sake. It is a recognition that approval, retrieval, orchestration, and policy evaluation will scale differently, fail differently, and need different operators. The prototype is good because it makes those seams visible early. It stops being enough once those seams need to carry real load.

This is the last post in the series, but it also ties back to the first one. The PRD starts with enterprise reliability requirements, and the code already reflects those priorities even where the implementation is still local or simplified. That is exactly how I would want an architecture prototype to behave.