Page 88 - CyberSecurityEssentialsEnglish

Sycophancy: When the System Agrees More Than It Should
Sycophancy is the tendency of a system to go along with the user’s opinion or assumptions
instead of offering an independent assessment. It can appear when the user frames the

question in a loaded way: “Isn’t this clearly the better option?” or announces the desired
decision and then asks for the analysis. Studies have shown that language assistants may
prefer an answer that matches the user’s position in some contexts, even where this

weakens truthfulness [8]. The tendency is not accidental. As Chapter One described,
assistants are tuned using human feedback, so they are optimized toward answers people
approved of, and approval and accuracy are not the same signal.

To reduce sycophancy, frame the question neutrally, separate facts from opinion, ask for the

strongest supporting argument and the strongest opposing argument, and specify what
could change the recommendation. You can also run a second analysis without revealing the

first opinion, give the criteria before the options, or use a human reviewer.

Do not rely on the question “what is your confidence level?” as though it were a calibrated
statistical probability. The number a model produces may be a linguistic estimate that does
not reflect the probability of being correct. It is better to ask for the assumptions, the

evidence, the part most exposed to error, the missing information, and what needs external
verification. A descriptive confidence of high, medium, or low can be used, provided its basis
is explained and it is not treated as a scientific measure.

Handling Uncertainty Without False Numbers

Uncertainty is not a single thing. It may come from a gap in the source, ambiguity in the
question, a conflict between pieces of evidence, a weakness in the model, or a change in the
world. Ask the system to identify the type of uncertainty rather than compressing it into a
number. For instance: “Classify the reasons for uncertainty into missing information,

assumptions, conflicting evidence, and model limits.”

When the work requires a decision, use a conditional recommendation: “If A holds, the first
option is more suitable; and if the cost exceeds B, the second option becomes better.” This

is more honest than a categorical recommendation built on incomplete information. The
conditions can then be turned into a data-collection plan: what is the smallest piece of
information that would most reduce the uncertainty?

In some specialized applications there are genuine probabilistic measures or calibration

scores, but they come from the design of the model and the evaluation system, not from a
general linguistic request. If the tool does not provide a documented measure, treat
expressions of confidence as descriptions of the reasoning rather than mathematical

probabilities. Do not write “95 percent confidence” in a professional report without a
statistical basis.

87