From Textual Criticism to AI Criticism: Evaluating Source Fidelity in AI-Generated Islamic Knowledge

Large Language Models are becoming a common point of entry for students and researchers working with premodern Islamic texts. They can locate a reported saying, explain a legal or theological position, and suggest a source within seconds. But an answer may be broadly correct while its evidence is not. A quotation can be slightly altered, a statement assigned to the wrong scholar, or a convincing reference given to a text in which it does not actually appear.

This paper examines this problem through a small comparative experiment focused on source fidelity. A set of questions will be selected from premodern Islamic materials in hadith, law, theology, and Qur'anic interpretation and submitted to several widely used language models under the same conditions. The references will first be checked against the relevant primary sources, providing a basis against which the models' answers can be assessed.

The analysis will distinguish between different kinds of reliability. Does the model give the right information? Does it attribute a statement to the right person or school? Does the cited work exist, and does it actually contain the material attributed to it? When the model presents words as a quotation, how closely do they correspond to the source? Particular attention will be given to references that sound credible but cannot be verified.

These questions matter especially for Islamic textual traditions, where the provenance of a statement is often inseparable from its scholarly value. The study therefore suggests that methods familiar from textual criticism and source verification can also help us evaluate AI-generated scholarship. Rather than treating accuracy as a single measure, it proposes source fidelity as a distinct category for assessing the use of LLMs in the study of premodern texts.