Page 115 - CyberSecurityEssentialsEnglish

If the context is “The sun rises from the ...”, the probabilities might be: east 0.52, horizon
0.25, today 0.15, and equation 0.08. These figures are a teaching illustration, not a
measurement of any particular model. Greedy decoding takes the most probable token

every time. It may increase consistency, but it can produce repetitive or overly conservative
text in open tasks.

FIGURE 3-3: THE PROBABILITY DISTRIBUTION AND THE CHOICE OF THE NEXT TOKEN

What to notice: Settings reshape the set of candidates or the rule of selection, but they do

not turn an output into a fact and they do not add a new source.
Source: Prepared by the author based on the principles of decoding and nucleus sampling in
[5].

Controls over Token Selection: Temperature, Top-p, and Top-k

Temperature controls how spread out the probability distribution is before the next token is

chosen. Low values concentrate the probability around the more likely candidates, which
makes outputs more consistent and more cautious and suits tasks that require precision and
disciplined wording. High values flatten the distribution and give less probable candidates a

greater chance of appearing, which may increase variety and creativity but also raises the
chance of outputs that are less accurate or less coherent. Temperature adds no new
knowledge to the model; it changes only how a choice is made from the distribution the

model already produced.

Top-k restricts the choice at each step to a fixed number k of the most probable tokens, then

redistributes the probabilities inside that set and samples from it. This helps exclude very
weak candidates, but it can be rigid, because the appropriate number of candidates differs

from one context to another.

Top-p, also known as nucleus sampling, selects the smallest set of tokens whose cumulative
probability reaches the value p. The size of the candidate set therefore changes

114