Page 380 - CyberSecurityEssentialsEnglish

TABLE 8-9: SELECTED GENERATIVE RISKS AND THE INSTITUTIONAL CONTROL

Risk What it looks like in practice The institutional control

Isolate content from instructions
Prompt A document instructs the system
and restrict tools and
injection to ignore its rules.
permissions.

Sensitive Data from a connected source Access control before retrieval,
disclosure appears in an answer. and no secrets in prompts.

Vendor review, version pinning,
An extension or component
Supply chain and removal of unused
changes or is compromised.
components.

Insecure output Generated text is executed or Schema validation, escaping, and
handling stored without validation. allowlists before execution.

Excessive The system can act more widely Least privilege, approval gates,

agency than the task requires. and rate and cost limits.

Source vetting, index integrity
Data or model Misleading content is injected
checks, and logging of retrieved
poisoning into a knowledge base.
passages.

Governing Agents and Systems With Executive Capability
The level of risk changes when a system moves from producing text to carrying out actions:

sending an email, updating a record, making a booking, running code, transferring a file, or
coordinating several tools. This pattern is often called agentic AI. Chapter Five sets out a
ladder of five levels running from a plain workflow to an agentic system, and the distinction

it draws is the one that matters here: a system that decides its own next step needs
institutional controls that a fixed sequence does not. The ability to plan and act across
successive steps means that an error may cascade before a human being sees it.

Published risk lists for agentic applications include hijacking an agent’s goal, misuse of tools,

misuse of identity and permissions, memory poisoning, insecure communication between
agents, cascading failure, exploitation of trust, and rogue agents [6]. Such a list does not
mean agents are unusable; it means that the controls suited to a simple chatbot are not

sufficient for a system able to act on the organization’s behalf.

379