Page 136 - CyberSecurityEssentialsEnglish

3.8 How Do We Evaluate a Tool in Practice?

◆ Key idea: local testing is not a single demonstration.

It is a protocol that includes representative and edge cases, repeated runs, blind
comparison, a classification of errors, and the cost of review and operation.

The Reference Evaluation Set and the Rubric
Build a local reference evaluation set, sometimes called a golden set. It must represent the

real distribution of the work, not easy examples only. Add ordinary cases, edge cases,
incomplete cases, badly formatted cases, and cases that have no answer, so that you can
test abstention.

Write the rubric before you see the results. It may cover factual correctness, coverage of the
requirement, fidelity to the source, language, format, time, and verifiability. Use scores with

clear descriptions rather than “I liked it” and “I did not.” When comparing, hide the name of
the tool and shuffle the order of the outputs to reduce bias.

FIGURE 3-7: THE TOOL EVALUATION CYCLE BEFORE AND AFTER ADOPTION

What to notice: An adoption decision is provisional, because the model, the interface, the

data, and the prices change. Make re-evaluation part of the process.
Source: Prepared by the author based on the principles of holistic evaluation [8].

135