Page 44 - CyberSecurityEssentialsEnglish

In a field experiment conducted with consultants at a large consulting firm, Dell’Acqua and
colleagues found that participants using AI completed about 12 percent more tasks, worked
about 25 percent faster, and produced work of higher quality on tasks that suited the tool.

On a task deliberately chosen to fall outside its capabilities, however, participants using AI
were roughly 19 percentage points less likely to reach the correct answer, because they

relied on its response without recognizing that the task was different [9]. The lesson is that
the same tool can be highly useful in one task and misleading in another.

This unevenness occurs partly because some subjects are well represented in training data,
while local or rare information is less common. A tool may perform well on a language task
yet struggle with calculation or with following many constraints. One application may also

use search or a calculator while another, built on the same model, does not. There is
therefore no fixed list of tasks that AI always performs well.

A small condition can also change a task’s difficulty. Summarizing a report may be easy, but
requiring the summary to preserve every legal exception raises the standard substantially.
Drafting a message may go well, but strict adherence to specialized terminology may expose

weaknesses. It is better to divide the task and test its components: facts, calculations,
wording, format, and sources. This reveals where a model is suitable and where a rule,
calculator, or expert is needed.

FIGURE 1-7: THE JAGGED TECHNOLOGICAL FRONTIER
What to notice: A task that seems easy to a person may require precision or local knowledge
the model lacks, while the model succeeds at a broad language task that appears more
difficult.

Source: Prepared by the author based on the concept of the jagged technological frontier
[9].

43