RiseHarness iteratively improves terminal-agent harnesses with weights fixed
It studies failed versus successful runs of the same task and adapts shared interaction mechanisms, not model weights.
Stories per day · 30 days
2 RSS
Relatedagent-harness1benchmark1coding-agents1commentary1fallback1long-horizon1model-routing1software-quality1terminal-agents1typed-decision-models1windows1
14 stories, newest first · one card per story
Stories tagged #reliability
Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
A new test finds typed decision models often mishandle fallback rules even without attacks.
Gremlin Foresight AI finds system weaknesses and verifies the fixes
Gremlin launched Foresight AI to find reliability risks, propose fixes, and retest them.
2
HERA co-evolves agent harnesses to abstain on infeasible tasks
An evolved harness raised abstention accuracy from 61.7% to 83.3% and transferred to 19 other LLMs.
When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Models
MOTIVE uses multi-view self-verification so vision-language models accept reliable answers and rethink uncertain ones.
Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
Expert-validated 1,042-item benchmark finds LLMs never exceed 0.45 free-text correctness on Colombian law, with about half of cited legal norms wrong or nonexistent.
Effective Resistance and Graph Neural Network Reliability in Tissue-Specific Interactomes
Effective resistance in 24 tissue interactomes mostly tracks inverse degree, limiting GNN reliability signals.
ControlScope: Workflow Revision and Reliability in LLM Agents
ControlScope finds only small, mixed gains from revising more of a running LLM-agent workflow.
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
IndicBankBench finds banking language-model assistants pass all three trials on only 43.7–58.2% of cases.
Bristol researchers say medicine already knows how to handle black boxes and AI could learn from it
Bristol researchers propose a drug-style dossier for testing medical AI reliability before clinical use.
1
Your Agent Aced the Task. Will It Do It Again?
IBM Research Hugging Face post examines whether LLM agents that succeed at a task once will reliably succeed again.
E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning
E2A-Bench, a 969-query financial chart reasoning benchmark, finds VLMs fail evidence-to-action consistency, with fine-tuning amplifying BUY:SELL bias 4-6x.
Cross-Model Agreement as a Deployment-Time Reliability Signal for Automatic Polyp Segmentation
Referee-Based Quality Estimation flags unreliable polyp segmentations at inference without ground truth, reaching ROC-AUC 0.960 with SegFormer-B0 referees.
Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Audit of eight cybersecurity LLM benchmarks shows evaluation pipeline choices can swing scores by over 80 points and reshuffle most model rankings.