Page 82 - CyberSecurityEssentialsEnglish

TABLE 2-11: DIAGNOSING THE OUTPUT INSTEAD OF ASKING FOR A GENERAL IMPROVEMENT

The visible defect Diagnostic question Possible adjustment

The answer is Are the purpose and the audience Add the decision required and the
generic clear? audience.

Are the source and the missing- Say use the material only, and
Invented information
value policy defined? write not stated.

The format keeps Did you show the format or use a
Add an example or a schema.
changing schema?

Did you define the relationship
The tone is
and the audience with an Add a short example of the tone.
unsuitable
example?

Did you define the priority and the Set a limit, a structure, and an
The result is too long
length? order.

Wide variation Was the prompt designed around Add varied evaluation cases and
across cases one example? compare.

Evaluation with Measures and Rubrics
When comparing two versions of a prompt, do not use a single question such as “which is
better?” Build a rubric that describes the criteria and the levels of performance. For example:

accuracy, coverage, fidelity to the source, clarity, and feasibility. A scale from 1 to 5 can be
used, with a description for each level. The description matters more than the number,

because it reduces differences of interpretation between reviewers.

Separate the criteria that can be checked automatically from those that need human
judgment. The number of fields, the length, and the validity of JSON can be measured
programmatically. The suitability of a tone or the strength of an explanation needs reviewers

or a human sample. Combining the two gives a more accurate picture than relying on a
model to evaluate itself.

In a simple A/B test, run the two versions on the same cases with the same settings as far as
possible, and hide the name of the version from the reviewer where you can. Record the

average performance and the number of serious failures, not the average alone. A version
may improve in most cases yet fail in a sensitive one, and that alone can be enough to
prevent adoption.

Pay attention to run-to-run variation. The model may produce different outputs for the same

request, especially in creative tasks. Test more than one run when consistency matters, or

81