Systematic Prompt Framework for Qualitative Data Analysis: Designing System and User Prompts

Abstract

<p>Prompt engineering has become an important aspect in optimizing the performance of large language models (LLMs) in diverse applications. This research proposes a systematic framework for system and user prompts by utilizing few-shot learning, chain-of-thought reasoning, role play and iterative refinement. The proposed framework was evaluated on open source LLMs, Llama, Gemma, and Phi, running on local machines to underscore their capability to enhance LLMs' outputs for qualitative data analysis for interview transcripts about security and privacy issues of gamification. Utilizing local LLMs eliminates concerns related to data leakage and privacy, making this approach particularly suitable for organizations that have privacy concerns with publicly available LLM solutions like ChatGPT, Gemini, DeepSeek etc. The LLM output demonstrated improved accuracy, consistency, and scalability in addressing security and privacy concerns with gamification. The validation using manual analysis with NVivo indicates less than 5% error margin for frequency analysis.<br></p>