ICALL-BENCH tests LLM interprocedural reasoning on real binaries
Original title: ICALL-BENCH: Benchmarking LLM Interprocedural Reasoning in Binary Programs
On 50 C/C++ programs, five models’ verifiable success ranged from 23% to 81%, and more information access did not reliably help the hardest cases.
Researchers introduce ICALL-BENCH to test whether LLMs and agents can recover indirect-call targets, used as a probe of interprocedural semantics in compiled binaries. The set covers 50 real-world C/C++ programs, averaging 74,746 lines of code and 941 functions, across complexity levels and optimization settings. Evaluation spans model-only reasoning, static tool-augmented agents, and Internet-enabled agents. The authors report uneven results and frequent verifiable failures, and say deeper interprocedural reasoning remains the main bottleneck.
Unconfirmed: Success rates are the authors’ reported figures and are not independently verified here.
- Programs average 74,746 lines of code and 941 functions.
- Three regimes compare model-only, static-tool, and Internet-enabled agents.
- Program-level verifiable success rates ranged from 23% to 81%.
- Extra information access brought limited gains on the hardest cases.
From the original article
1 min read at arxiv.orgRecent benchmarks show that LLMs achieve strong performance in both function-level binary understanding and end-to-end binary reverse engineering tasks, but whether LLMs can reason over interprocedural semantics in compiled binaries remains unclear.
By Shenglong Yao · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.