Post-pilot evidence lab

# AI quiz item-analysis checklist

An item-analysis evidence matrix and post-pilot review protocol for interpreting facility, discrimination, option behavior, omissions, timing, comments, and source validity without letting one statistic decide an AI-generated question's fate.

Published16 October 2026Reading time11 minEvidenceItem-analysis evidence matrix and post-pilot review protocolAuthorThe AI Quiz Lab

Use item analysis after a pilot or real delivery to find AI-generated questions that behave unexpectedly. Review sample size and administration conditions first; then inspect response counts, facility or percent-correct, discrimination, option choices, omissions, time, comments, and source validity together. No single statistic should automatically delete or approve an item.

## Key findings, verified 16 October 2026

- Moodle's current quiz statistics report documents quiz- and question-level psychometric analysis, including attempts, facility, discrimination, discriminative efficiency, and response-frequency views.
- Penn State's guide treats reliability, item difficulty, item discrimination, and distractor information as connected evidence that should not be examined independently.
- Very easy or very hard items can show low discrimination even when they intentionally test mastery, prerequisites, or an extreme level of challenge.
- A negative or weak discrimination result is a review trigger, not proof that the learner or item is at fault; a wrong key, ambiguous wording, or a defensible unkeyed answer can cause the pattern.
- Post-pilot statistics cannot repair an unsupported answer key, inaccessible delivery, biased sample, or invalid construct.

## What is AI quiz item analysis?

Item analysis reviews response behavior for each question after participants have attempted the quiz. For AI-generated items, it complements prepublication source review: one asks whether the content is defensible, while the other asks how the item performed in one sample and delivery context. The item ID, answer key, scoring rule, route, and quiz version must remain stable enough to connect the observed statistics to the delivered item.

## Item-analysis evidence matrix

EvidenceWhat it can showQuestion to askUnsafe conclusionAttempt countAmount of observed dataIs the sample large and complete enough?A small pilot proves stabilityFacility / percent correctHow often the keyed response earned creditWas this difficulty intended?High facility means a bad itemDiscriminationRelationship between item and total performanceDoes the item distinguish in this sample?Low discrimination always means deleteOption frequencyWhich choices participants selectedAre distractors functioning as intended?Unchosen options are automatically wrong to keepOption discriminationWho chose each optionIs a distractor attracting stronger participants?A surprising option proves participant errorOmissionSkips or blanksWas the item inaccessible, confusing, or late?Blank always means no knowledgeTime and pathExposure and response conditionsDid routing or latency affect opportunity?All participants saw the same itemComments and source auditQualitative defect evidenceIs the key, wording, or source wrong?Statistics override documented content defects

## Reproducible post-pilot review

- Freeze the delivered version. Preserve the quiz, answer key, scoring, route, settings, and stable item IDs used in the pilot.
- State the intention. Define the construct, item purpose, and expected difficulty before looking at results.
- Describe the sample. Record attempts, completion, exposure, and exclusions; do not merge materially different forms without justification.
- Recheck the content. Verify the source, key, accepted variants, points, and feedback again.
- Export defined measures. Collect facility, discrimination, option frequency, omissions, and timing where supported, preserving each platform's calculation definition.
- Triangulate. Review measures together and flag contradictions rather than selecting a preferred statistic.
- Inspect delivery evidence. Review learner comments, accessibility reports, routing logs, and device problems.
- Make a recorded decision. Classify each item as retain, monitor, revise, re-key, exclude from score, or retire—with a reason.
- Change one controlled version. Record the revision and every affected key, score, or feedback rule.
- Repilot. Test the revision before treating improved behavior as durable.

## Worked example: a negative discrimination warning

An application item is answered correctly by 42% of the pilot, but higher-scoring participants choose one distractor more often than the key. The source audit shows that both options are defensible because a qualification was dropped from the stem. The correct action is not to blame strong learners or merely replace the distractor. Apply the assessment's correction policy, restore the qualification, document any re-keying or score exclusion, and pilot the revised item.

## Analysis specification

For every stable item ID, return attempts, exposure count, maximum score, mean item score, facility, discrimination or discriminative efficiency, option frequencies, option discrimination where available, omissions, median response time, comments, source-check result, and review decision. Preserve the calculation definition used by the platform. Flag insufficient data instead of ranking unstable estimates.

## Publication blockers

- The delivered quiz version or key cannot be reconstructed.
- Routed participants had unequal exposure but the analysis treats attempts as equivalent.
- A known content or scoring defect remains inside the statistics without correction or annotation.
- Measures from different definitions or platforms are compared as if identical.
- Small, selected, or incomplete samples are presented as generalizable.
- One cutoff automatically deletes or approves an item.
- A revision has no version record or follow-up pilot.

## What this checklist does not prove

Item analysis describes performance in the observed sample. It does not by itself establish reliability, validity, fairness, accessibility, causal learning, or predictive value. It also does not justify high-stakes use of an AI-generated assessment without qualified psychometric and subject-matter review.

## Sources, verification date, and corrections

This guide was verified on 16 October 2026 against Moodle's current [quiz statistics report](https://docs.moodle.org/en/Quiz_statistics_report), Penn State's current [Guide to Item Analysis](https://teaching.psu.edu/pdf/GuideToItemAnalysis.pdf), and the University of Washington's [item-analysis guidance](https://www.washington.edu/assessment/scanning-scoring/scoring/reports/item-analysis/). Those sources support the reported measures and the need to interpret them in context. The evidence matrix, ten-step review, worked example, analysis specification, and blockers are original AI Quiz Lab assets. Report changed guidance or reproducibility problems through the [corrections process](https://bestaiquizgenerator.com/corrections).

## Next checks

Return to the [answer-key checklist](https://bestaiquizgenerator.com/blog/ai-quiz-answer-key-checklist) for every flagged item, then use the [distractor guide](https://bestaiquizgenerator.com/blog/ai-quiz-distractors) and [feedback rubric](https://bestaiquizgenerator.com/blog/ai-quiz-feedback-rubric) when revising content.

---

Canonical: https://bestaiquizgenerator.com/blog/ai-quiz-item-analysis