Studying illustrated texts and videos in primary classrooms: Learning outcomes, cognitive load, studying processes, and individual differences

Abstract

Despite the prevalent use of multimedia in primary education, there is limited understanding of how different learning material modalities, such as videos and illustrated texts, affect learning outcomes in classroom settings. Understanding these effects is crucial for educators to tailor instructional strategies that meet the needs of students to facilitate learning. The modality effect is a phenomenon where people learn better from pictures and narration (e.g., videos) than from pictures and written text (e.g., illustrated texts). This dissertation comprises three empirical studies that investigate the modality effect in primary school classrooms. Specifically, it examines how illustrated texts versus videos impact key learning metrics such as retrieval, retention, transfer, and cognitive load, among fifth and sixth graders in Finnish basic education. Additionally, it assesses how studying rate, studying strategies, and reading skills affect these outcomes.

Study I explored the modality effect in a learner-paced classroom setting with a crossover experiment. In the experiment, 54 fifth graders learned science topics from both illustrated texts and videos. They completed pretests, posttests, and delayed tests to assess cognitive load and learning outcomes. The combination of the within-subjects design with linear mixed-effects analysis quantified individual difference factors. The results showed that the video condition outperformed the illustrated text condition in retention, delayed retention, cognitive load, and efficiency measures, although no significant differences were found in transfer tasks. These results show that the modality effect can occur in an authentic primary school classroom environment. Previous studies have often explained that the learning benefits of videos are based on the illustrated narrations causing less cognitive load than illustrated texts. Contrary to this traditional explanation, cognitive load did not emerge as a significant predictor of learning outcomes in Study I, challenging the widely held assumption that videos' learning advantages stem from their ability to reduce cognitive load. This discrepancy points to the need for an alternative explanation for the modality effect.

Motivated by the above findings, Study II aimed to explore factors other than cognitive load for explaining the outperformance of videos. Past research, including Study I, has predominantly focused on learning outcomes without collecting information on the learning process. Therefore, Study II analyzed 97 screen recordings that were captured on children’s iPads during the learning phase of Study I. The analysis examined how children learn from videos and illustrated texts. In particular, studying rates and strategies were assessed; that is, how fast and in what order the children studied the materials. The results demonstrated that videos were studied faster than illustrated texts, and this faster studying predicted improved learning outcomes and lower cognitive load. With both modalities given the same amount of study time, faster studying allowed students to spend more time reviewing the material, providing a plausible explanation for the superior learning outcomes in the video condition. Analysis of the studying strategies provided an explanation for faster studying in the videos: children tended to watch the videos linearly at their designed rate with minimal navigation, and this rate of the videos was faster than the children’s reading speed, particularly for weaker readers. Also, the children reviewed mostly by navigating to the start of the video and watching it again from start to finish. While the studying rate in videos was highly homogenous, the studying rates varied widely with illustrated texts. Also review strategies were more varied with illustrated texts. While the videos progressed on their own linearly forward unless interacted with, the illustrated texts required children to actively navigate among slides. This added effort and decision making may explain the higher cognitive load in the illustrated texts.

The slower studying rate in illustrated texts suggests that children’s weak reading skills may explain the modality effect, since videos do not require reading. However, Studies I and II lacked direct measurement of reading skills to confirm this.

Study III was designed to address the above gap by investigating the roles of decoding ability and reading comprehension in learning from videos versus illustrated texts. Additionally, Study III aimed to enhance the ecological validity of previous studies by allowing students access to the instructional materials during the posttest, better reflecting typical classroom conditions. A within-subjects crossover experiment involving 109 fifth to sixth graders replicated the results of Study I: videos led to better retention and lower cognitive load. Decoding ability and reading comprehension were both found to be positive predictors of learning outcomes in both modalities. Importantly, an interaction between reading skills and modality was observed: videos provided greater benefits for delayed retention among students with weaker reading abilities. Interestingly, despite prior research suggesting that poor decoding increases cognitive load in text-based materials, there was no significant interaction between reading skills and modality with respect to cognitive load. This suggests that the cognitive load benefits of videos lie elsewhere.

Together, the results challenge the prevailing explanation that the modality effect arises from avoiding cognitive overload. Cognitive load did not explain learning differences between modalities, overall levels of cognitive load were well below capacity limits, and decoding ability did not explain cognitive load differences between modalities. Instead, the results point to alternative explanations and suggest that the modality effect is shaped by higher-level, context-dependent mechanisms. In the classroom context, a key mechanism appears to be the guided and linear nature of studying videos, which standardizes the studying rate and coverage across the content.

The main results of this dissertation have practical value for primary education. Videos can lead to better learning outcomes, lower cognitive load, and faster studying compared to illustrated texts. Moreover, they can promote more equitable learning by inducing more uniform studying, reducing variation in learning outcomes, and supporting students with weaker reading skills. Whereas most previous studies on the modality effect were conducted in highly controlled settings outside the classroom, this dissertation demonstrates these effects in authentic classroom contexts, offering practical information for educators. As producers of learning material often provide educators with both video and illustrated text versions of their content, the findings help teachers make informed decisions in everyday classroom situations.