Page 325 - CyberSecurityEssentialsEnglish

7.6.1 Extended Case Study:
From a Convincing Review to an Auditable Evidence Map
A methodological note: this is a composite teaching case for the purposes of explanation,
not an account of any particular event. It is written to show how one moves from a research

question to an evidence matrix and a synthesis that can be inspected.

An institutional research team began with a broad question: “does AI improve the
productivity of customer service teams?” In a first attempt, the team asked a tool to

generate a short review and received a coherent text stating that automation raises
productivity and lowers costs. The text was convincing, but it combined chatbots,
recommendation systems, summarizing tools, and writing assistants, and it treated

productivity as the same thing as satisfaction, response speed, and ticket volume. The
problem was not the phrasing; it was that the question itself was not researchable.

The team started again. It narrowed the question: what is the effect of using a conversation-
summarizing assistant on ticket closure time and documentation accuracy in medium-sized

service centers? It then defined productivity as closure time and the proportion of tickets
not reopened within seven days. It built a term table covering customer support, ticket
resolution, summarization assistant, and agent productivity, with Arabic and English

alternatives.

After searching an academic database and official report sites, the team entered the records
into an evidence matrix. It emerged that most sources discussed the general adoption of AI,

while only a few measured conversation summarizing. It also became clear that some
reports had large samples but were issued by companies selling the same solutions; they
were therefore treated as contextually relevant but limited in independence. Conversely,
some peer-reviewed studies were of high quality but concerned small experimental

environments unlike the organization’s reality.

AI helped turn the abstracts into fields, propose reasons for differing results, and highlight
missing values. But it did not make the inclusion decision, did not write the final quality

judgment, and did not attribute a finding to a source before the text was opened. Every
important claim in the final paragraph traced back to a row in the matrix and a location in
the source. When the manager asked why an immediate wide rollout was not

recommended, the team could show that the evidence pointed to a probable benefit in
reducing documentation time, but did not on its own establish an improvement in service
quality or a fall in reopened tickets.

The evidence map produced a more mature recommendation: a limited internal trial over

eight weeks, two primary indicators, a comparison group, an error log for the automatic
summaries, and human review of sensitive samples. That conclusion is less striking than the

324