Evaluate retrieval before polishing the answer

A fluent answer can conceal a retrieval problem. If the right document never reaches the model, a better-written prompt may only produce a more convincing unsupported response.

Start an evaluation plan at the data boundary, and work toward the answer.

Build a small question set

Choose questions that reflect actual work. For each one, record the relevant source passage, the expected access context, and what a satisfactory answer must include. Include questions that cannot be answered from the available documents.

A useful first set covers direct lookups, synonyms, questions that need two sources, outdated documents, and missing information. This is an evaluation fixture, not a benchmark claim. Its value depends on how well it represents the users and documents involved.

Inspect retrieval separately

Ask whether the expected source appears in the retrieved results. When it does not, inspect ingestion, document versioning, chunk boundaries, metadata filters, and ranking before changing generation instructions.

This runnable Python example computes source-level recall for one question. Production evaluation should define relevance more carefully, particularly when several passages from one document play different roles.

def recall_at_k(expected: set[str], retrieved: list[str], k: int) -> float:
    if not expected:
        raise ValueError("This metric requires at least one relevant source")
    if k < 1:
        raise ValueError("k must be positive")
    hits = expected.intersection(retrieved[:k])
    return len(hits) / len(expected)

print(recall_at_k({"procedure-a", "procedure-b"},
                  ["procedure-a", "unrelated", "procedure-b"], 2))
# 0.5 for this illustrative fixture; not a measured system result.

Recall alone does not tell you whether the ranking is useful, whether the source is current, or whether the answer cites the right passage. It is one diagnostic signal.

Separate retrieval quality from answer quality

Use a scorecard that lets you locate a failure rather than compressing everything into a single number.

Ragas documents context precision as a way to examine relevant context in ranked retrieval results, and faithfulness as support for generated claims in the supplied context. These are distinct diagnostics. An answer can faithfully repeat a stale source, so document validity still needs its own check.

If an automated judge helps with review, compare its judgments with human-labeled examples. Keep its model and prompt version alongside the results; otherwise a changed judge can look like a changed application.

Test access before generation

Enforce permissions in the retrieval and application layers. Asking the model to hide an unauthorized passage is too late: the content has already crossed the boundary.

Test the same question under different user roles. Check both the retrieved text and the final answer. Permission checks should also cover document updates, cached results, and any tool that retrieves additional context.

Define the missing-evidence behavior

An internal assistant needs a useful way to say it cannot support an answer. Agree on whether it should ask a clarifying question, link to the closest relevant source, or direct the user to a person.

Evaluate factual claims and citations separately. A link to a real document is not enough if the cited passage does not support the answer.

Keep a regression set

Retest important questions after changing the model, prompt, ingestion, or ranking. Save the configuration alongside each run so a change in results can be investigated. Keep sensitive examples in an appropriately controlled environment.

References

• Ragas: Context precision — ranked retrieval evaluation.

• Ragas: Faithfulness — support for answer claims in retrieved context.

• OWASP: Prompt Injection explains risks from untrusted inputs, including content encountered through external sources. Retrieval provides context; it does not make that context trustworthy instructions.