Page 137 - CyberSecurityEssentialsEnglish

TABLE 3-12: COMPONENTS OF A LOCAL EVALUATION PROTOCOL

Component Practical application

A representative sample covering ordinary, edge, incomplete, and
The cases out-of-scope cases, with a source or a reference answer where

possible.

Three or more runs for generative cases to measure variation,
Repeated runs
recording the version and the settings.

Hide the name of the tool, shuffle the order of the outputs, and use
Blind comparison
a fixed rubric.

Error Facts, sources, numbers, extraction, formatting, language,

classification abstention, privacy, or execution.

Response time, review time, rework, integration, training, and
Time and cost
service failures.

The adoption A minimum threshold, rejection gates, an approval owner, a scope

decision of use, and a re-evaluation date.

Measures and the Interpretation of Differences
The choice of measure begins with the type of output. In classification you may use precision,

recall, or F1, but accuracy can conceal the failure of a rare class. In extraction, measure the
accuracy of each field and the proportion of documents that pass without intervention. In
retrieval systems, separate retrieval from generation. In translation and professional writing

you need a human or specialist rubric that measures meaning, terminology, and tone, not
word overlap alone.

Do not reduce everything to a single average. Present the distribution of errors and the worst
failures, because two tools may reach the same average: one stable with small errors, and

another excellent most of the time but failing badly in a few cases. If the impact of failure is
high, the worst case and the rate of abstention may matter more than the average.
Small differences may be noise. Use the same cases for all tools, repeat runs for probabilistic
outputs, hide the names, and keep individual judgments before group discussion. A

confidence interval can be offered when the sample is large enough, but the more important
point for a general reader is not to declare a winner on the basis of three cases or a one-

point difference.

Create an error taxonomy: wrong source, wrong number, missed condition, formatting,

language, excessive abstention, or incorrect execution. The classification guides the remedy.
A score says the system is weak; the type of error says which layer needs repair.

136