Developing a Data-Driven Feedback System to Support Students’ Multiple-Source Learning Processes in LearnNet

Abstract

In today's digital educational environment, students are expected to find, evaluate, and synthesize information from multiple online resources. The ability to do so is termed "Multiple Source Comprehension" (MSC), and it remains one of the most difficult skills for students to develop. To master these complex tasks, students require continuous teacher support and instructional scaffolding, which is time-consuming for teachers to deliver individually. While artificial intelligence systems can provide some support through automated mechanisms, such as providing instant feedback, transforming students’ digital trace records into supportable feedback is a technically challenging task. As such, the purpose of this research is to extend the LearnNet platform’s ability to provide automated feedback through new process-oriented feedback formats and to evaluate the system-wide capability of generating AI-based interventions using real-world school contexts.

The primary development contribution of this study includes designing and integrating the Entire Task Feedback System. This system collects raw time-series event log data that represent the activities of students and transforms these logs into a set of quantitative indicators that measure the learning processes of students. Additionally, the full system, including all feedback modalities, was tested in live school trials in Finland.

To establish the factual accuracy of the system, its usefulness as a source of actionable feedback, and comprehensibility, an independent evaluation framework was developed that integrated metrics from the LLM as a Judge with interaction data collected via the user interface. During live trials, an important instructional bottleneck known as the performance inversion trap was identified. Early baseline engines provided high clarity feedback; however, they also had low actionability and delayed executions. A controlled offline experiment was implemented to determine how different models and reasoning budget constraints impacted feedback quality and latency. The results indicate that moving to the newer generation model substantially improved the tradeoff between feedback quality and latency. Specifically, 53.8% of generations delivered high-tier actionable scaffolding while maintaining structurally safe factual accuracy and reduced average round-trip latency from 22.38% to a classroom-acceptable 5.19 seconds per request. Finally, post-task surveys from the pilot group indicated that students perceived the grounded scaffolds as both clear and reliable and displayed no apparent symptoms of systemic AI dependency. Due to the relatively limited scope of the pilot, further research is needed to replicate these results on a larger scale.