Page 125 - CyberSecurityEssentialsEnglish

When a question arrives, the system may rewrite the query, then use lexical, semantic, or
hybrid search. The results are reranked, the passages to be sent to the model are selected,
and the answer is generated with links or page numbers. Two separate stages must be

evaluated: did the system fetch the right evidence, and did the model use it faithfully without
distortion or overreach? The quality of this stage depends on the retrieval models, the

semantic representations, the reranking, and the standardized evaluation of embeddings, as
the literature on dense passage retrieval, sentence embeddings, and embedding
benchmarks makes clear [18][19][20].

The quality of retrieval also depends on governance. Document permissions must be

respected, a user must be prevented from retrieving material they are not entitled to see,
the index must be updated when a version changes, malicious instructions inside files must
be handled, and the passages that supported an answer must be logged. Grounding in

sources does not automatically equal correctness; the source itself may be outdated or
inappropriate.

FIGURE 3-6: THE FULL PATH OF A TRUSTWORTHY RETRIEVAL SYSTEM

What to notice: Evaluate retrieval and generation separately, and do not assume that the
presence of a vector store or a visible citation is enough.
Source: Prepared by the author based on [6] and [15].

How Do We Measure a Retrieval System and Secure It?
Retrieval-augmented generation should be evaluated in at least two stages. In the first, the

retrieval stage, the basic question is: did the system find the correct passage or document
and show it among the leading results? Three measures help answer it. Recall@k asks
whether the required passage appeared among the first k results. Precision@k measures the

proportion of relevant results among them. Mean reciprocal rank measures how high the
first correct result sits, which matters when the order of the results matters. A general

124