Design and Study of Evidence-Linked AI Support for Reviewing Multi-Source Student Synthesis

Abstract

Reviewing student synthesis tasks in multi-source reading environments places a demanding cognitive load on teachers. When students work with several sources of varying quality, the synthesis text alone does not reveal which sources were used, whether critical ideas were covered, or whether the student’s reading strategy was systematic. This thesis presents the design and evaluation of an augmented teacher panel for the LearnNet multi-source reading environment, intended to make student process data and AI-generated coverage assessments accessible to teachers alongside the synthesis text.

The panel was designed and implemented following a Design Science Research approach and evaluated through a qualitative case study with three teacher participants, who reviewed student cases in both manual and augmented conditions using a think-aloud protocol. Four research questions examined transparency of student understanding, alignment between AI and teacher judgments, panel efficiency and usefulness, and technical feasibility of the underlying large language model service.

Results showed that traceability and the specific student text passage that grounds it were the primary determinant of perceived AI adequacy. The efficiency benefit of AI-generated summaries was mediated by teacher domain familiarity. Narrative group reports were more interpretable for instructional planning than quantitative aggregation displays. The LLM service was technically feasible, with 55 of 74 (74.3%) session calls completing within 30 seconds.

A cross-cutting finding was that all three teachers engaged with the AI support in a verification-first mode, checking AI claims against student text evidence before accepting them. This pattern is interpreted as evidence that effective teacher-facing AI must be designed around conditional trust and inspectable reasoning - the teacher-in-the-loop principle - rather than accuracy alone.