Understanding Large Language Model Performance in Vulnerability Detection: A Systematic Analysis

Large Language Models (LLMs) have emerged as powerful tools for software vulnerability detection, yet their decision-making processes remain opaque and highly sensitive to prompt design. This paper presents a systematic study on how different forms of auxiliary information - metadata-derived program semantics, retrieval-augmented generation (RAG), and model-derived insights such as in-context learning and chain-of-thought reasoning - affect both detection performance and model interpretability. Through controlled experiments on three state-of-the-art LLMs (LLaMA-3.3-70B, Qwen-3-32B, and GPT-OSS-120B), we analyze not only accuracy-related metrics but also reasoning behaviors using structured prompting inspired by Explainable Artificial Intelligence (XAI) principles. By exposing consistent reasoning patterns and failure modes, this work provides empirical insights toward more transparent, trustworthy, and explainable LLM-based vulnerability detection systems.