LLM-Assisted Qualitative Data Analysis: Security and Privacy Concerns in Gamified Workforce Studies

Abstract

Large language models (LLMs) have transformed textual or qualitative data processing and analysis by automating and enhancing interpretive accuracy, particularly in complex areas like cybersecurity, ethics, and compliance. This study examines the effective-ness of local LLMs in analyzing qualitative research using the data gathered from the case study on "perspectives on security and privacy issues associated with the introduction of gamified workforce studies". The research presented in this paper utilized 23 interview transcripts to evaluate three popular LLMs, namely LLaMA, Gemma, and Phi, running on a local infrastructure. We observed that LLaMA focuses on practical data security, Gemma on regulatory compliance, and Phi on ethical transparency and trust-building. By combining these models, researchers can gain a more comprehensive understanding of the complex implications of gamification in workforce studies. Local LLMs provide the added benefit of enhanced data privacy and security by processing sensitive data entirely within a controlled environment. This study explores the system and user prompts that can improve the interpretive accuracy of various qualitative research approaches, such as thematic analysis, frequency analysis, impact level analysis, sensitivity analysis, and disclosure analysis, demonstrating the potential of local LLMs for qualitative analysis for sensitive data. This study recommends the usage of LLMs for the initial stage of the qualitative analysis process to enhance the efficiency and effectiveness of subsequent completely manual or software-assisted manual analysis.