CodeTD: Topology of Attention Detects Hallucinations in Code LLMs
CodeTD detects hallucinations in code LLMs before execution by analyzing topological patterns of attention maps, outperforming recent baselines.
CodeTD applies topological data analysis (TDA) to code LLM attention maps to quantify prompt-generation mismatch as a pre-execution correctness signal. Experiments cover HumanEval, MBPP, BigCodeBench, and MultiPL-E across 5 programming languages and 10 code LLMs up to 34B parameters. The method outperforms recent baselines and transfers between coding benchmarks, helping catch code that fails the task or embeds security vulnerabilities.
- First TDA-based pre-execution hallucination detector for code LLMs
- Evaluated on HumanEval, MBPP, BigCodeBench, and MultiPL-E benchmarks
- Tested across 5 languages and 10 code LLMs up to 34B parameters
- Outperforms baselines and transfers across coding benchmarks
Full article124 words · extracted from arxiv.org · click to collapse
As AI-code assistant tools become widespread, automatic assessment of the correctness of generated code becomes a significant challenge. Code LLMs are prone to hallucinations, which may lead to code that does not solve the required problem, or even to code with severe security vulnerabilities. In this paper, we introduce CodeTD -- the first approach to pre-execution assessment of code correctness based on topological data analysis (TDA) of Code LLMs' attention maps. Our method quantifies prompt-generation mismatch using topological patterns of attention maps. We carry out experiments with common benchmarks (HumanEval, MBPP, BigCodeBench, MultiPL-E), 5 programming languages and 10 Code LLMs of size up to 34B parameters. The experimental results show that the proposed method outperforms recent baselines. Moreover, CodeTD is transferable between coding benchmarks.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.07779