CodeTD: Topology of Attention Detects Hallucinations in Code LLMs
CodeTD detects hallucinations in code LLMs before execution by analyzing topological patterns of attention maps, outperforming recent baselines.
CodeTD applies topological data analysis (TDA) to code LLM attention maps to quantify prompt-generation mismatch as a pre-execution correctness signal. Experiments cover HumanEval, MBPP, BigCodeBench, and MultiPL-E across 5 programming languages and 10 code LLMs up to 34B parameters. The method outperforms recent baselines and transfers between coding benchmarks, helping catch code that fails the task or embeds security vulnerabilities.
- First TDA-based pre-execution hallucination detector for code LLMs
- Evaluated on HumanEval, MBPP, BigCodeBench, and MultiPL-E benchmarks
- Tested across 5 languages and 10 code LLMs up to 34B parameters
- Outperforms baselines and transfers across coding benchmarks
- Products
- CodeTD
From the original article
1 min read at arxiv.orgAs AI-code assistant tools become widespread, automatic assessment of the correctness of generated code becomes a significant challenge. Code LLMs are prone to hallucinations, which may lead to code that does not solve the required problem, or even to code with severe security vulnerabilities.
By Daria Voronkova · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.