Page 313 - CyberSecurityEssentialsEnglish

TABLE 7-10: THE SHORT PROTOCOL CARD

Protocol element The question it answers

Question and scope What will we answer, and what will we not cover?

Search sources Where will we search, and why these sources?

Inclusion and exclusion
What enters and what is excluded?
criteria

How many reviewers, and how will disagreements be
Screening plan
resolved?

Extraction fields What information will we take from each source?

Quality and bias
Which instrument or questions suit the design?
appraisal

Synthesis plan Narrative, map, quantitative pooling, or a mixture?

Managing amendments How do we document a departure from the plan?

7.4.2 Risks of Bias and Certainty in the Evidence

It is not enough to ask whether a source is peer-reviewed or official. The sharper research
question is: where could the methodological error lie, and what would its effect on the result
be? The points of risk differ by design. A trial is vulnerable to weak randomization and

attrition, an observational study to confounding, a qualitative study to shallow coding or
absent reflexivity, and an institutional report to the definition of an indicator or the
incentives of the body producing it.

When synthesizing a body of evidence, it helps to think in terms of certainty. The most widely

used system for this is GRADE, short for Grading of Recommendations, Assessment,
Development and Evaluation. In its language, confidence may fall because of risk of bias,
inconsistency, indirectness, imprecision, or publication bias.

It may sometimes rise in non-randomized studies where there is a large effect, a dose-

response relationship, or residual bias working against the observed result [22][32]. We do
not need to apply the full formal system in every field, but its language helps a reader

understand why we say the evidence is strong or limited.

312