Risk Detection in E-commerce with LLMs: Annotation Challenges and Lessons from Real-World Business News

Abstract

The growing complexity of e-commerce supply chains has amplified the need for effective risk monitoring systems. While Large Language Models (LLMs) have demonstrated potential in various domains, their application to real-world risk detection in e-commerce remains underexplored. This study introduces a novel, manually annotated dataset of 121 business news articles covering five major e-commerce-related steel companies, ArcelorMittal, Tata Steel, POSCO, NLMK, and ThyssenKrupp, annotated using the Cambridge Risk Taxonomy. We evaluate the performance of two advanced LLMs in detecting and classifying risks across multiple categories using few-shot prompting and semantic similarity-based example selection. Our results show that LLMs can approximate human annotation with moderate micro F1-scores and high coverage, though challenges remain in recognizing complex Geopolitical risks and avoiding overgeneralization. The findings provide actionable insights into the potential and limitations of LLMs for automated, domain-aware risk monitoring, laying the groundwork for future applications in supply chain risk management.