Page 147 - CyberSecurityEssentialsEnglish

? Question to consider: which factor should carry the weight?
If the style of the answers improved while the accuracy of the cited page fell, would you
count the new version as better? What weight should attribution carry in this particular
task?

3.12 Practical Lab: Comparing Three Tools on a Common Basis
↻ The goal of this lab
To produce a decision card that can be reviewed, not to declare an absolute winner.
Suggested time: 90 to 120 minutes, carried out on non-sensitive or anonymized data.

1. Choose one task of low or medium risk, and define the verb, the output, the success
criterion, and the impact of failure.
2. Create 12 to 20 evaluation cases that represent reality, including edge cases, incomplete

cases, and one case where the system should abstain.
3. Choose three tools from the same category, or document clearly how the categories
differ.

4. Fix the instructions, the sources, and the settings as far as possible, and record the version
and the date.
5. Run the cases more than once where the output is generative, and save the outputs

without the product names.
6. Score blind using a rubric of four to six criteria, and classify the errors rather than
recording a score alone.

7. Compute the time to an acceptable task, including review and rework, and do not stop
at response time.
8. Apply the mandatory gates, then the seven weighted factors, and write a decision with a

defined scope.
9. Define what would trigger suspension or re-evaluation: a model update, a policy change,

or recurring errors.
10. Save the decision card, the reference data, and the settings so that the test can be
repeated later.

146