Beyond Content Overlap: Evaluating AI-Generated Clinical Summaries in Finnish SOTE Social Care Using Novel Longitudinal and Risk-Weighted Metrics

Abstract

Standard content overlap metrics such as BERTScore and ROUGE are structurally inadequate for evaluating AI-generated summaries of longitudinal social care records: they measure textual similarity to a reference but cannot assess temporal coherence, trajectory fidelity, or severity-proportionate error impact. This thesis develops and evaluates a systematic automated evaluation framework for AI generated social care summaries in the Finnish SOTE setting.

The framework was evaluated on 30 fully synthetic patient records across four complexity tiers, including seven adversarially designed edge cases targeting documented AI failure patterns. GPT-4o-generated summaries were assessed through a multi metric automated pipeline; a 15-case gold standard subset received independent dual annotation with inter-rater reliability measurement. Two novel candidate metrics are introduced: the Longitudinal Context Preservation Metric (LCPM), targeting temporal and trajectory fidelity, and the Risk-Weighted Quality Score (RWS), which penalises errors multiplicatively by clinical severity. Six dynamic safety rules assess whether summaries address safety-critical information present in each source record. A prototype review interface, Vigil, was developed as a practical deliverable for Mediconsult Oy.

RWS correlates significantly with human-rated Clinical Usefulness (r = 0.684, p = 0.005, N = 15), with stability confirmed across four alternative penalty weight configurations (r range 0.684–0.744). A baseline comparison showed ROUGE-1 (r = 0.822) and BERTScore (r = 0.768) produce stronger correlations with Clinical Usefulness on the gold standard subset, attributable to a reference quality artefact that does not replicate in deployment contexts without corrected reference summaries. BERTScore correlated moderately with human-rated Clarity (r = 0.520, p =0.047). LCPM showed the expected complexity tier ordering but did not reach statistical significance at N = 15 (r = 0.378, p = 0.165). The safety rules achieved a 76.6% overall pass rate; seven cases carried critical safety failures across 102 detected errors. The findings indicate that AI summaries require structured human review before clinical deployment. RWS and the safety rule framework provide an initial empirical basis for risk-stratified review triage.