Most question problems come from a small set of repeat mistakes: answer options that echo the question, matching pairs with identical values, and numbers or letters typed into answer options. This article lists each problem, why it matters, and how to fix it, with a checklist to run before publishing.
It applies to every question type used in Develop, Assess and Survey: Single Answer, All Which Apply, Multiple Answers, Ordered, Match, Image Map and Free Text. Because every type is scored all-or-nothing, one flawed option or pair can mark a knowledgeable User wrong, skew their Knowledge Rank and undermine trust in the results.
Answer option problems
Answer options should be short, distinct and free of anything that hints at the answer.
| Problem | Example | Why it matters | Fix |
| Option repeats the question | Q: ‘What is the purpose of a risk register?’ Correct option: ‘The purpose of a risk register is to record risks.’ | The matching words point straight to the answer, and the option becomes long and awkward | Answer only: ‘To record and track identified risks’ |
| Numbers or letters typed into options | ‘1. Report it’, ‘A) Escalate’, ‘Option 3 – Ignore it’ | The platform randomly lays out the options itself, so typed labels clash with the display and stop making sense if the order changes | Remove the prefixes and enter the plain text only |
| Duplicate or near-duplicate options | ‘Within 24 hours’ and ‘Within one day’ | Two options mean the same thing, so one is a free elimination, or both are correct | Make every option meaningfully different |
| Overlapping ranges | ‘0–5 days’, ‘5–10 days’, ‘10+ days’ | Day 5 and day 10 sit in two options | Use clean bands: ‘0–4’, ‘5–9’, ‘10 or more’ |
| Correct option stands out | The right answer is the longest, most detailed or only qualified option | Test-savvy Users pick it without knowing the content | Keep options similar in length, detail and tone |
| Absolute words in distractors | ‘Always’, ‘never’, ‘only’ in every wrong answer | Users learn that absolutes are wrong and eliminate them | Use absolutes only where the policy really is absolute, and in correct options too |
| Grammar gives the answer away | Q: ‘An employee should report an...’ with only one option starting with a vowel | The sentence only fits one option | Rewrite the stem so every option reads correctly (‘...report which of the following?’) |
| Trailing full stops and mixed capitals | Some options end in a full stop, some don't | Looks careless and can mark out the correct option | Pick one style and apply it to every option |
Match and Ordered question problems
Every pair in a Match question must have exactly one correct partner, and every item in an Ordered question must have exactly one correct position.
| Problem | Example | Why it matters | Fix |
| Identical values in column 2 | ‘Fire alarm → Call 999’ and ‘Medical emergency → Call 999’ | The User can't tell which ‘Call 999’ goes where, so a correct answer can be marked wrong | Make each value unique (‘Call 999 and evacuate’, ‘Call 999 and give first aid’) or split into separate questions |
| Identical or reversed items | ‘GDPR → GDPR Definition’ and ‘CIA Definition → CIA’, or the term and definition swapped in one pair | The pairs test nothing, or the columns stop being consistent | Keep one column for terms and one for definitions throughout |
| More than one valid match | Two definitions that both fit ‘Phishing’ | Several arrangements are defensible but only one scores | Tighten the wording so each item fits one partner only |
| Pair gives itself away | ‘Data Protection Officer → The officer responsible for data protection’ | Repeated words solve the pair without knowledge | Describe the role without reusing its name |
| Mixed categories | Some pairs match a term to a definition, others a role to a person | Users can match by category rather than understanding | Keep every pair the same kind of relationship |
| Too many pairs | Eight or more pairs on a mobile screen | Hard to read and drag on a phone, and one slip fails the whole question | Keep to around four to six pairs |
| Ordered steps with no fixed order | Steps that can happen in any sequence, or two steps that happen together | Only one order is accepted, so a valid alternative fails | Only use Ordered where the sequence is set by policy or process |
Question wording problems
The question itself should ask one clear thing that a User could answer before seeing the options.
| Problem | Example | Why it matters | Fix |
| Negative wording | ‘Which of these is not a reason not to share your password?’ | Double negatives trip up people who know the answer | Ask positively; if a negative is unavoidable, use one and put it in capitals (‘Which is NOT...’) |
| Two questions in one | ‘Who approves expenses and how long does it take?’ | Someone who knows half the answer still fails | Split into two questions |
| Vague or subjective wording | ‘What is the best way to handle a complaint?’ | ‘Best’ is opinion unless policy defines it | Anchor it: ‘According to the complaints policy, what should you do first?’ |
| Trick questions | Correct answer depends on a small detail hidden in the wording | Tests reading speed, not knowledge, and frustrates Users | Test the knowledge directly |
| Missing context | ‘What is the limit?’ | The User can't tell which limit is meant | Name it: ‘What is the gift-acceptance limit for a single gift?’ |
| Instruction doesn't match the type | Multiple Answers question says ‘select all that apply’, or the stated number differs from the number of correct options | The scoring rule shown to the User contradicts the question | Let the type's built-in instruction do the work, and check the count matches |
| Internal jargon or acronyms | ‘What does the RCSA cover?’ | New starters or other teams fail on vocabulary, not knowledge | Spell out acronyms on first use unless the acronym itself is being tested |
| Content that will go out of date | Named individuals, prices, dates or version numbers | The question becomes wrong without anyone noticing | Avoid them, or set an expiry so the question is reviewed |
Correct answer and distractor problems
Each question needs a correct answer that is clearly right and wrong options that are believable.
| Problem | Example | Why it matters | Fix |
| More than one correct option in Single Answer | Two options are both true in practice | Users who pick the other true answer are marked wrong | Make the distractor clearly wrong, or change to All Which Apply |
| Wrong option marked as correct | Correct answer set on the wrong option after editing or copying | Every right answer scores as wrong and Knowledge Rank drops for everyone | Re-check the correct-answer setting after any edit |
| Silly or implausible distractors | ‘Throw the laptop in the river’ | Users eliminate them instantly, so a four-option question becomes a three-option guess | Base distractors on real mistakes or common misconceptions |
| Too few options | Two options on a Single Answer question | A 50% chance of guessing correctly | Aim for three to five options |
| Free Text answer too strict | Correct answer ‘VA102’ with case sensitivity on, or ‘10’ when Users may type ‘ten’ or ‘10 days’ | Correct knowledge is marked wrong because Free Text needs an exact match | Turn on ignore case, use numbers only, or say the expected format in the question |
| Image Map target too small or unclear | The correct area is a few pixels, or two similar elements appear | Users know the answer but can't tap it on a phone | Use a large, clearly distinct target area and a clear image |
| Survey options with a right answer | Survey question written as a knowledge test | Surveys have no correct answer, so the data is misleading | Move it to a Develop or Assess topic |
Formatting and explanation problems
Small presentation issues make questions look unreliable and can change how they read on mobile.
| Problem | Example | Fix |
| Spelling, grammar and mixed spellings | ‘organisation’ in the question and ‘organization’ in an option | Proofread and use consistent spelling throughout |
| Stray spaces and line breaks | Extra spaces or line breaks pasted in from Word or Excel | Paste as plain text, then check the preview |
| Formatting pasted from other tools | Bullets, smart quotes or symbols that render as odd characters | Retype special characters and check the preview |
| Overly long questions or options | A paragraph of policy text in the question | Keep the question to one or two sentences; move background to the explanation |
| Missing or weak explanation | No explanation, or one that only says ‘The answer is B’ | Explain why the correct answer is right in one or two sentences, so the PQE teaches as well as scores |
| Explanation refers to option labels | ‘Option C is correct because...’ | Refer to the answer by its text, not its position |
| Inconsistent units and formats | ‘£1,000’, ‘1000 GBP’ and ‘one thousand pounds’ across options | Use one format for numbers, currency and dates |
Embedded video and pre-learning document problems
Videos and pre-learning documents only help if every User can open them, and the questions test what they actually cover.
| Problem | Example | Why it matters | Fix |
| Question tests content not in the material | The pre-learning PDF covers the 2025 policy; the question asks about the 2026 change | Users who did the reading are marked wrong | Write questions from the attached material, and update both together |
| Answer only in the video, with no text | The key fact is said once in the audio, with no captions or transcript | Excludes Users who are deaf or hard of hearing, and those who can't play sound at work | Add captions or a short transcript, and show key facts on screen |
| Large or desktop-only document | A 60-page PDF, or a file that only opens in desktop Word or Excel | Slow or unreadable on a phone, so Users skip it | Link to the relevant page or section, or summarise it in a short mobile-friendly document |
| Video or document gives the answer away | The question appears right after a video that shows the answer on screen | Tests short-term memory, not understanding | Ask the Users to apply the content to a scenario rather than repeat it |
| Content that goes out of date | A video showing an old version of a system screen or a superseded policy | Users learn the wrong process, and the question may contradict it | Date the material, set an expiry on the topic, and replace content when the process changes |
| External or third-party link | A public YouTube video or supplier website | Adverts, autoplay suggestions or blocked sites at work, and the owner can remove it at any time | Host the content internally where possible, or check the customer's network allows it |
| Unclear what to watch or read | ‘See attached’ with three files and no instructions | Users don't know which part matters | Say what to look for: ‘Watch the first two minutes on reporting a breach’ |
Before you publish: checklist
Run through these checks for every question before it goes live.
- ☐ The question asks one thing, positively, and makes sense before reading the options
- ☐ No option repeats the wording of the question
- ☐ No numbers, letters or ‘Option 1’ prefixes typed into options
- ☐ Every option is distinct, with no duplicates or overlapping ranges
- ☐ Options are similar in length, style and punctuation
- ☐ Match questions have unique values in both columns and one valid partner per item
- ☐ Ordered questions have one fixed, policy-based sequence
- ☐ The correct answer is set on the right option(s) and the count matches the instruction
- ☐ Distractors are plausible mistakes, not jokes
- ☐ Free Text answers allow for case and format where appropriate
- ☐ The explanation says why the answer is right, without referring to option positions
- ☐ Spelling, grammar, units and formats are consistent
- ☐ The question has been previewed on mobile