Page 45 - CyberSecurityEssentialsEnglish

How Can You Test Whether a Tool Fits Your Task?

Success on one example is not enough. Build a local test set that includes common and
difficult cases, using the language and format of the real workflow. Before testing, define
the success criterion, unacceptable errors, and the person responsible for review. If outputs

vary across runs, repeat important cases and record the differences.
Do not rely on one average score. A system may succeed on short documents and fail on
long ones, work well in Modern Standard Arabic and poorly with a local dialect, or succeed

most of the time but fail on rare, high-impact cases. Compare the tool with the current
workflow: the aim is to improve the whole process, not to produce an impressive
demonstration.

TABLE 1-8: DESIGNING A LOCAL TEST SET FOR A REAL TASK

Test element Question Example

Do the cases represent real Short and long messages, dialect,
Coverage
use? and local terminology.

Which examples reveal Missing information or two similar
Edge cases
weaknesses? tables.

What is the minimum No invented figures, and 95% of
Success criterion
acceptable result? essential fields extracted.

Which errors are Changing a person’s name or giving
Impact of error
unacceptable? unsupported financial advice.

Is the output stable across
Repeatability Run important cases three times.
several runs?

Does it outperform the Time, quality, rework, and reviewer
Comparison
current method? satisfaction.

↻ Try it yourself: A ten-case test
Choose one low-risk task and collect ten real cases after removing sensitive data. Set a

clear standard for each case, then compare the output with the current method. Record
the cases in which the system fails; they are more informative than the average success
rate.

“Do not ask, ‘Is the model powerful?’ Ask, ‘Does this task, with this data and in this context,
fall within its capabilities?’”

44