Source fidelity lab
How to turn a PDF into an AI quiz without losing source fidelity
A reproducible traceability audit for checking whether every generated question, answer and explanation is supported by the PDF.
- Published
- 24 August 2026
- Reading time
- 12 min
- Evidence
- 10-item traceability audit
- Author
- The AI Quiz Lab
To turn a PDF into an AI quiz safely, require every question, correct answer and explanation to map to a specific passage in the source. Treat an item without a traceable passage as unsupported, even when it sounds plausible.
Key findings, 24 August 2026
- A source upload is not proof that the output stayed inside the source.
- Traceability must cover the stem, correct answer and explanation, not the answer alone.
- A ten-item calibration set reveals boundary failures before a full quiz is generated.
What is source fidelity?
Source fidelity is the degree to which a generated quiz preserves the facts, scope, uncertainty and meaning of its supplied document. A faithful item can be traced to evidence in the PDF without adding outside knowledge or turning a qualified statement into a certainty.
The risk is not limited to fabricated facts. A quiz can distort a source by dropping a condition, combining unrelated passages, selecting a defensible but not uniquely correct answer, or writing an explanation that goes beyond the text. NIST recommends documenting knowledge limits and evaluating factual validity in generative AI systems, while UNESCO emphasizes human oversight in educational uses of generative AI. See the NIST Generative AI Profile and UNESCO guidance.
What should the PDF prompt require?
Reusable source-bound specification
Use only the attached PDF. Create 10 questions that cover its main ideas. For each item, return the question, options, one best answer, a short explanation, the page number and the exact passage that supports the answer. Preserve qualifications such as “may,” “often” and “under these conditions.” If the PDF does not support an item, label it unsupported instead of using outside knowledge.
How does the 10-item traceability audit work?
- Select a short PDF section you can review manually.
- Generate exactly ten items using the specification above.
- Locate the cited passage for each item.
- Score the stem, answer and explanation separately.
- Reject the batch if any answer is unsupported or if more than one answer is defensible.
- Record recurring failure types, revise the specification once, and repeat with a new ten-item set.
| Audit field | Pass | Fail |
|---|---|---|
| Passage exists | Quoted or precisely located | Missing, vague or outside the PDF |
| Meaning preserved | Conditions and uncertainty remain | Claim broadened or certainty inflated |
| Answer uniqueness | One option is best from the passage | Two options are defensible |
| Explanation boundary | Explains only with source evidence | Adds unsupported context |
| Coverage | Items sample distinct key ideas | Items repeat one easy section |
Worked traceability example
Source passage: “The pilot ran for six weeks. Attendance increased in four of the six groups, but the report did not measure learning outcomes.”
Faithful question: What did the pilot report? Best answer: Attendance increased in four groups. The item preserves both the measured result and its limit.
Unfaithful question: How much did the pilot improve learning? The PDF contains no learning-outcome result, so any numeric or directional answer would be invented.
What failures should block publication?
- Boundary leak: the item uses common knowledge that is not in the PDF.
- Qualification loss: “may” becomes “will,” or an observed association becomes causation.
- Fragment mismatch: the citation exists but does not support the intended answer.
- Answer leakage: wording in the stem makes the answer obvious.
- False coverage: repeated questions create the appearance of breadth.
What are the limitations?
Page citations can break when a PDF has no reliable text layer, uses scanned images or has complex columns. Optical character recognition can introduce its own errors. A passed traceability audit establishes support, not educational validity, fairness or an appropriate difficulty level. Those require separate review.
Use this alongside the question quality checklist, the six-part prompt specification and the AI Quiz Lab methodology.
Sources and verification
This guide was verified on 24 August 2026. Facts are linked to primary or institutional sources. The worked models and checklists are original AI Quiz Lab evidence assets, designed to be repeated with any generator.
If a source changes or you find an error, use our correction path. The publication-neutral next step is to test the framework on a small, low-stakes quiz before using it at scale.