Page 172 - CyberSecurityEssentialsEnglish

the user to make the model set aside its own rules, through role play, a hypothetical framing,
an unusual language, or a request broken into innocuous-looking pieces. The two are often
confused because both end in the system doing something it should not, but the controls

differ: injection is reduced by isolating and distrusting content, whereas jailbreaking is
reduced by limiting what the system is able to do at all, and by checking outputs rather than

relying on the model to police itself.

Improper output handling appears when a program takes the model’s text and places it
directly into a database, a page, a system command, or code. Treat model outputs as
untrusted input: check the structure and the values, escape text, allow only a defined list of

commands, and never execute code that has not been reviewed.

Excessive agency means the system can do more than the task requires. It may hold read,
write, and delete access, or select many tools without approval. The remedy is least privilege,
fewer tools, rate limits, approval before important actions, and an immediate ability to stop.

Controls of this kind, placed around a model rather than inside it, are usually called
guardrails, and their strength is that they hold even when the model itself is deceived.
There is also supply chain risk: an extension, a model, a library, or a connector may change,

be compromised, or collect data. Do not trust an integration because it is well known. Review
the source, the updates, and the permissions, test it in a separate environment, and remove

extensions you do not use.

For the non-technical user, the practical rule is this: do not give a new tool access to all of
your email or files merely to try a feature. Begin with a non-sensitive file or a limited folder,
read the permissions, revoke access after the test, and do not approve automatic execution
whose details you cannot see.

+ For the advanced reader: the risks of agentic systems
When a system does more than answer, and instead plans, calls tools, keeps memory,
and communicates with other agents, additional risks appear: goal hijacking, tool misuse,
misuse of permissions and identity, poisoning of memory and context, insecure
communication between agents, and approval fatigue. The practical controls are: a
separate identity for the agent, least privilege, reviewable memory, a limited tool list, a
log for every action, and explicit approval before any consequential step. These controls

align with current thinking on the risks of agentic applications [20].

171