Retrieval Only Works If the Knowledge Path Stays Honest

A lot of agent architectures talk about retrieval as if it were a side feature. That is backwards. If the knowledge path is weak, stale, or impossible to reason about, the rest of the system is just better-packaged guessing.

The current KB path is simple, and that simplicity is doing two jobs

The current implementation keeps things deliberately small. ingest.py defines a handful of sample IT articles, embeds them with SentenceTransformer("all-MiniLM-L6-v2"), and inserts them into a local Chroma collection at startup.

docs = [a["content"] for a in KB_ARTICLES]
ids = [a["id"] for a in KB_ARTICLES]
metas = [{"title": a["title"]} for a in KB_ARTICLES]
embeddings = model.encode(docs).tolist()

collection.add(
    documents=docs,
    embeddings=embeddings,
    metadatas=metas,
    ids=ids
)

That is a perfectly reasonable prototype choice. It gives you a real retrieval path without forcing you to stand up a separate ingest service on day one.

There is also a useful design decision hiding here: the embedding model is initialized once and then handed into the tool layer through init_tools(). The retrieval function is not reloading model state on every call.

_collection = None
_embed_model = None

def init_tools(collection, embed_model):
    global _collection, _embed_model
    _collection = collection
    _embed_model = embed_model

The retrieval path is short enough to inspect, which is a strength

The query path in search_knowledge_base() is straightforward: embed the user question, query Chroma for the top two matches, and convert the top distance into a crude confidence score. If the KB is empty, the tool returns a structured error so the agent can fall back cleanly.

@tool
def search_knowledge_base(query: str) -> dict:
    if _collection.count() == 0:
        return {"error": "KB_EMPTY", "content": "Knowledge base is currently empty.", "confidence": 0.0, "low_confidence": True}

    embedding = _embed_model.encode([query]).tolist()
    results = _collection.query(
        query_embeddings=embedding,
        n_results=2,
        include=["documents", "metadatas", "distances"],
    )

flowchart TD
    Q["User question"] --> E["Embed query"]
    E --> S["Chroma similarity search"]
    S --> R{"Any matches?"}
    R -->|no| Empty["Return KB_EMPTY / no results"]
    R -->|yes| Score["Take top distance"]
    Score --> Confidence["confidence = 1 - distance"]
    Confidence --> Gate{"Above threshold?"}
    Gate -->|yes| Result["Return article content"]
    Gate -->|no| Review["Flag low confidence for review"]
                    

This is a respectable first cut, but only because the limitations are visible

The confidence model is useful mostly because it reveals its own weaknesses

I like that the app does not blindly trust retrieval. Low confidence can trigger human review, which is a mature pattern. But the implementation is still early. The code comment says cosine distance below this = low confidence, while the logic actually computes 1 - distance and compares that derived score to the threshold. That mismatch is small, but it is exactly the kind of detail that turns into confusion later when teams start tuning thresholds in production.

distance = results["distances"][0][0]
confidence = max(0.0, 1.0 - distance)
low_confidence = confidence < CONFIDENCE_THRESHOLD

return {
    "content": results["documents"][0][0],
    "title": results["metadatas"][0][0].get("title", ""),
    "confidence": round(confidence, 3),
    "low_confidence": low_confidence,
}

This is the kind of snippet I wanted more of in the original draft. It lets the reader judge the retrieval policy directly instead of trusting a prose summary. And once you can see the exact confidence calculation, the limitations become obvious in a productive way.

The real problem is not vector search, it is knowledge freshness

The PRD calls for a scheduled freshness pipeline with chunk versioning and rollback. The prototype does none of that yet. It ingests a static Python list. That is fine for showing retrieval behavior, but it does not address the real support problem: KB articles drift all the time. VPN instructions change. Device policy changes. Approval paths change. A stale helpdesk agent is worse than a slow one.

If I were hardening this path, I would stop thinking about ingest as “load documents into Chroma” and start thinking about it as a release process for knowledge. That means each refresh needs a version tag, rollback target, and retrieval-quality telemetry attached to it. The PRD is right to call this out, because embedding drift is an operational problem, not just an ML problem.

What I would not ship in the retrieval layer

I would not ship a retrieval system where confidence is a thin wrapper around one distance score and the source corpus is a hardcoded list in application memory. That is enough to exercise the agent loop, but it is not enough to defend an answer when someone asks why the system retrieved this article instead of that one.

I also would not ship a knowledge path without versioned ingest. If retrieval quality drops after a refresh, you need a rollback target, not a theory. Otherwise every KB update becomes an untracked experiment running against production users.

What the production version should add

• Pull source documents from a real KB system instead of a hardcoded array.
• Chunk and version documents so embedding sets can roll forward and backward safely.
• Run freshness jobs on a schedule rather than during user session startup.
• Track retrieval quality by document version so regressions are visible.

Once retrieval is weak, the rest of the architecture pays for it

The agent loop, the fallback behavior, and even the approval system all depend on retrieval quality. If the KB is empty or confidence is weak, the app either needs a human to step in or a safe fallback to take over. That is why the knowledge service in the PRD is its own boundary. It is not just a data store. It is a reliability surface.

In the next post I’ll move one layer outward and look at the security controls around the agent. Retrieval grounds the model, but it does not protect the system on its own. That is where prompt injection checks and scoped tool binding come in.