Assessment integrity lab

# AI quiz randomization integrity test

A reproducible shuffle-safety matrix and invariance test for keeping answer IDs, scores, feedback, branches, media, accessibility, and meaning stable across randomized quiz attempts.

Published4 October 2026Reading time10 minEvidenceShuffle-safety matrix and randomized-attempt invariance testAuthorThe AI Quiz Lab

Run repeated quiz attempts with a fixed seed when possible, then verify that shuffling changes presentation without changing the answer key, score, feedback, accessibility, or meaning. Do not shuffle ordered scales, dependent question groups, “all of the above” choices, or any item whose wording refers to option position.

## Key findings, verified 4 October 2026

- Moodle supports random question selection and answer-option shuffling as separate controls.
- Shuffling is useful only when every randomized version remains equivalent.
- Ordered responses, matching pairs, anchored choices, and dependent stems need explicit exclusions.
- A reproducible seed or stored presentation order is necessary to investigate disputes.
- Randomization may reduce copying opportunities but does not prove security, fairness, reliability, or validity.

## What is a quiz randomization integrity test?

A quiz randomization integrity test compares a canonical attempt with many shuffled attempts. It checks invariants: the facts and relationships that must remain unchanged even when question or option order changes. These include stable answer IDs, total points, blueprint coverage, branch behavior, feedback, media associations, labels, keyboard order, and announced meaning.

## Shuffle-safety matrix and randomized-attempt invariance test

ElementUsually safe?Required invariantCommon failureIndependent questionsYesSame blueprint and scoreA prerequisite appears after a dependent itemStandard MCQ optionsSometimesStable answer ID controls gradingThe key remains “B” after the answer movesOrdered scaleNoSemantic orderAn agreement scale becomes meaninglessMatching optionsPlatform-dependentEach pair remains linkedDuplicate labels or broken pair IDsAll or none of the aboveNoGrouped reference stays meaningfulThe referenced option set changesMedia and follow-upSometimesResource and feedback stay attachedAn image or rationale follows the wrong item

## Reproducible integrity test: nine steps

- Inventory every randomized layer. Record bank selection, question order, option order, variables, and branches.
- Classify each item. Mark it shuffle-safe or fixed, with a written reason.
- Create a canonical attempt. Store stable question IDs, answer IDs, scores, feedback, and resource links.
- Generate test attempts. Produce at least 20 randomized attempts, or enumerate every possible order for a small set.
- Compare the invariants. Confirm total points, blueprint coverage, correct answer IDs, and branch behavior are unchanged.
- Check attached content. Verify labels, media, equations, explanations, and feedback stay with the right item.
- Test access paths. Run keyboard and screen-reader checks on at least two materially different orders.
- Preserve the evidence. Save the presented order or random seed with each attempt.
- Investigate by ID. Trace any score difference using stable IDs, never display letters or positions.

## Worked example: the answer key stores a letter

A four-option question stores “B” as correct. After shuffling, the correct answer text moves to D, but the grader still accepts the second displayed option. Some participants receive a wrong score depending on order. The repair is to grade against a stable answer ID and save the presented answer-to-position mapping for audit.

## When an item must stay fixed

Keep an item fixed when position carries meaning. That includes ordered rating scales, procedural sequences, matching sets whose labels are not unique, dependent question groups, and wording such as “the option above.” Grouped options such as “all of the above” and “none of the above” are also poor shuffle candidates. Published MCQ design guidance warns that these grouped options can increase cognitive load or let a participant answer with only partial information.

## Source-bounded randomization QA prompt

Using only the canonical item records, answer IDs, score rules, branch rules, resource links, feedback records, and randomized attempt logs supplied below, compare each attempt with the canonical version. Report any change in correctness, total points, blueprint coverage, feedback, branch behavior, labels, media associations, keyboard order, or announced meaning. Identify items by stable ID, not display position. Return “not tested” when the evidence does not cover an invariant.

## Publication blockers

- Total points or blueprint coverage changes across equivalent attempts.
- A correct answer is keyed to a letter or display position instead of a stable answer ID.
- A label, image, equation, hint, explanation, or feedback message follows the wrong item.
- An ordered scale, procedural sequence, dependent group, or positional reference is shuffled.
- Keyboard order or screen-reader meaning changes in a way that blocks or misleads participants.
- The system cannot save or reconstruct the presented order for a disputed attempt.

## What this test does not prove

This test establishes presentation invariance for the attempts and states tested. It does not prove test security, deterrence of collaboration, psychometric equivalence, reliability, fairness, accessibility conformance, or validity. Preview the complete quiz and use qualified assessment and accessibility reviewers where results affect high-stakes decisions.

## Sources, verification date, and corrections

This guide was verified on 4 October 2026 against Moodle's current [Building Quiz documentation](https://docs.moodle.org/501/en/Building_Quiz), its [Multiple Choice question documentation](https://docs.moodle.org/en/Multiple_Choice_question_type), and the peer-reviewed [Educator's blueprint for developing high-quality multiple-choice questions](https://pmc.ncbi.nlm.nih.gov/articles/PMC9873868/). Those sources support separate question and option shuffling, selective fixed sections, previewing, option-level grading, and the warning against grouped options. The matrix, nine-step test, worked example, prompt, and blockers are original AI Quiz Lab evidence assets. Report changed guidance or reproducibility problems through the [corrections process](https://bestaiquizgenerator.com/corrections).

## Next checks

Use the [question-type decision matrix](https://bestaiquizgenerator.com/blog/ai-quiz-question-type-matrix) before enabling shuffling, test thresholds with the [score-band boundary audit](https://bestaiquizgenerator.com/blog/ai-quiz-score-band-test), and run the randomized experience through the [accessibility checklist](https://bestaiquizgenerator.com/blog/accessible-ai-quiz-checklist).

---

Canonical: https://bestaiquizgenerator.com/blog/ai-quiz-randomization-test