Page 33 - CyberSecurityEssentialsEnglish

Data is not always neutral. Collection methods determine what is represented and what is
missing, and older data may contain errors or practices we do not want to repeat. A large
volume of data is not enough; the data must also be relevant, accurate, current, and

representative of real use.

The objective must also be chosen carefully. If customer-service quality is measured only by
how quickly a case is closed, a system may learn to favor rapid closure over solving the

problem well. What we measure guides what the system attempts to improve.
During training, a model adjusts its internal values to reduce error on the target task. When
the trained model receives new text, an image, or other data and produces an output, it is

performing inference. The practical question is whether the training data resembles the
cases the system will face in actual use.

Common Ways Models Learn
Models can be trained in several ways. They may learn from examples with correct answers,

identify groups and patterns in unlabeled data, or learn through trial and reward when they
succeed. Some language models are trained to predict a part of the text that has been hidden
[6], while others are trained to predict the next token. Both approaches allow a model to

learn from very large amounts of unlabeled text without a person labeling each example.
Some models are first trained on broad datasets and later adapted to specific tasks. This
helps explain how one model can perform many language tasks, but it does not mean the

model knows your organization’s procedures or the latest regulations unless it is provided
with accurate, current sources.

One further stage explains a great deal about how a conversational assistant behaves. After
a model has learned to predict text, it is usually tuned further using human feedback: people
compare candidate answers, and the model is adjusted to produce the kind of answer

people preferred. The best known form of this is reinforcement learning from human
feedback, usually abbreviated RLHF. This is why an assistant follows instructions, keeps a
helpful tone, and declines certain requests, none of which follows automatically from

predicting the next word.

This stage is also worth understanding because of what it optimizes. The signal is human
approval, and approval is not the same thing as truth. A model tuned toward answers that
people liked may become agreeable in ways that do not serve the reader, which is the origin
of the tendency Chapter Two examines under the name sycophancy. Knowing where that

tendency comes from makes it easier to design around it rather than being surprised by it.

32