Page 41 - CyberSecurityEssentialsEnglish

1.7 What Does AI Not Guarantee?

◆ Key idea: A useful system does not guarantee correct results.
A system can be valuable without guaranteeing factual accuracy, currency, exact
calculation, understanding of unstated context, or confidentiality. Knowing these limits
helps us decide when a task requires a source, a calculation tool, or an expert.

First, an AI system does not guarantee factual accuracy. A generative model is designed to
produce coherent text, not to verify every sentence. When it lacks a source or detail, it may
hallucinate: produce an incorrect claim, number, or reference that sounds convincing.

Chapter Four discusses methods for verifying such outputs in detail [10].

Second, a system does not guarantee that information is current. A model’s knowledge may
stop at a certain date, known as its knowledge cutoff, or it may rely on an outdated source.

Even a tool connected to the internet may select a weak source. Information may have been
true in the past but no longer be current.

Third, a system does not guarantee accurate calculation merely because it can explain
mathematics. Language models process symbols and may make mistakes in long addition,
unit conversion, or percentage calculation. When a reliable number is required, use a

calculator, spreadsheet, or other calculation tool, then ask the model to explain the result.
Likewise, statistical similarity does not prove causation; a system may describe a correlation
as though it were a cause.

Fourth, a system does not guarantee understanding of implicit context or values. It may read
a policy but not know the organization’s unwritten culture, the sensitivity of a relationship,
or the effect of a decision on an individual. It may also reflect biases in the data or in the

objective’s definition. We should therefore ask not only, “Can it?” but also, “Should it, and
what would an error cost?”

Fifth, a system does not guarantee equal performance across groups or cases. It may work

well for one group of users, one dialect, or one document type and noticeably less well for
another, because the training data or the chosen objective did not represent them equally.
Results should therefore be examined group by group, not only as a single average score.

Sixth, a system does not guarantee confidentiality. Data entered into a tool may be stored,
processed outside the organization, or used in ways the user did not intend. Personal,

contractual, and internal information should be handled only in an approved environment,
and no more data should be provided than the task requires.

40