Learning Behaviour and Assessment Dynamics Before and After the Emergence of Large Language Models: A Cohort Comparison in an Introductory Object-Oriented Programming Course

Abstract

Large Language Models (LLMs) such as ChatGPT became publicly available in late 2022. These tools can write working code from a short description, which raises a direct question for programming education: has the way students work through a course changed since they arrived? This thesis examines that question using data from a single object-oriented programming (OOP) course at the University of Turku, taught in the same form in 2022 and again in 2025. The 2022 cohort took the course before LLMs were widely available, whereas the 2025 cohort took it afterwards. Thus, a natural comparison is formed between a pre-LLM and a post-LLM group of students.

The behavioural logs from the course's automated assessment platform, ViLLE, are analysed together with the course surveys. The behavioural data covers 248 students in 2022 and 230 in 2025, and includes every submission, the first and best score on each exercise, the submission timing, and the final examination score. The two cohorts are compared on engagement, tutorial performance, procrastination, and examination results using non-parametric tests with effect sizes. Furthermore, two exploratory machine learning analyses are added, such as a clustering of weekly learning trajectories and a feature importance analysis of which weeks best predict the examination.

It is found that the 2025 cohort submitted far fewer times per exercise, with stronger first attempts and less improvement through resubmission, while reaching almost the same best tutorial score. Of these differences, the reduction in submissions is the one robust finding, whereas the score differences are significant but small. Moreover, survey data revealed the two cohorts had similar prior experience and did not find the course any easier, yet the latter cohort scored lower on the examination, a comparison weakened by the change from an unsupervised examination in 2022 to a supervised one in 2025. Overall, the behavioural pattern is consistent with students completing tutorial exercises with outside assistance such as AI tools, although the observational design and the absence of any direct measure of LLM use mean that this cannot be confirmed.