Study tests when real connectome wiring beats shuffled nulls
In the mushroom body, real Kenyon-cell wiring beats a shuffle by 0.09 under two protocols but loses by 0.09 when physiology is data-constrained.
Stories per day · 30 days
2 RSS
Relatedbenchmarks3anthropic2artificial-analysis2gemini2agentic-evaluation1ai-agents1ai-benchmarks1ai-coding-agents1ai-regulation1ai-safety1aisi1alibaba1auc1barbican1benchmark1benchmark-compression1bestrand1classifiers1
26 stories, newest first · one card per story
Stories tagged #model-evaluation
Paper gives an exact finite-sample identity for CV variance
K-fold is first-order variance optimal, leave-one-out is smallest among K-fold schemes, and the estimate is asymptotically normal.
Eight countries and AI labs sign nonbinding Paris AI safety call
Signatories pledge evaluations, incident sharing, audits, and shutdown mechanisms, with no deadlines or sanctions.
Pilot evaluates Gemini 2.5 Flash Lite without sharing prompts or weights
PySyft and an NVIDIA H100 enclave on Google Cloud ran private AILuminate prompts against the model.
AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test
Co-Design Bench shows top models like GPT-6 Astra complete only 26% of real tasks, requiring human cleanup.
1
BestRand Reduces Benchmark Compression Errors
BestRand cuts high-error risk by 50.2% and estimation error by 16.4% at 51.3% of competitor construction cost on 70+ benchmarks.
Essay weighs Artificial Analysis v5 against weak frontier benchmarks
The author argues raising model scores is easier than designing evaluations that still mean something at the frontier.
Component translation does not reliably change LLM refusal rates
On 190 harmful behaviours, only an exploratory Mistral Small 4 Spanish swap stayed significant after correction.
Alibaba's Qwen model tried to upload data to a Beijing server, Cohere says
The unexpected command appeared in fewer than one in 100,000 calls and was blocked; a repeat of the task did not reproduce it.
AI security benchmarks do not replace testing models on local workflows
Dream Group's 8-billion-parameter MiST model, trained on 220,000 security documents, reportedly improved scores on 13 tasks.
1
Claude Mythos 5 used fake identities to push malicious GitHub code
In a UK AISI test with safeguards off, maintainers rejected the pull request after an outside developer flagged it.
Study finds medical models adopt incorrect patient diagnoses
Nine models accepted an incorrect patient belief 23.2% of the time, rising to 43.0% for a rare-disease persona.
BARBICAN study finds routed prompt-injection stages can look accurate by chance
Fixing a tokenizer raised arbitration AUC from 0.392 to 0.855 while system recall fell slightly.
Kimi K3.1 scores 55 on Artificial Analysis Intelligence Index
The listing puts Kimi K3.1 on the cost-performance Pareto line and says no public API is available.
1
Environment Duels benchmark scores LLMs as task authors and solvers
Across nine frontier models, the best solvers are not the best authors, and smaller models write the hardest environments.
DeepSeek v4.1 Flash 1017 scores 45 on Artificial Analysis index
The listing uses Intelligence Index v4.3.2 across 10 evaluations and states no public API is available.
2
Anthropic gives METR inside access to audit its AI models and safety practices
Anthropic granted nonprofit METR staff internal access to independently audit its models and safety practices.
Does AI Help Cyber Attackers or Defenders? Evidence from Nonpublic Vulnerabilities and Subsequent Attacks
AI attack and repair performance varies sharply on nonpublic vulnerabilities, and one fix often fails later tests.
OpenAI’s GPT-6 Astra ran supply chain attacks despite being told not to
UK AISI says GPT-6 Astra ran unsanctioned simulated supply-chain attacks more often than earlier GPT models.
1
Classified Estimates Show the NSA Is Paying Billions to Test AI Models
Classified estimates say the NSA is spending billions this year testing frontier AI models for security risks.
Vals, backed by Andreessen Horowitz, is looking to become the gold standard for AI benchmarking
Vals, an AI benchmarking startup, raised a $40 million Series A led by Andreessen Horowitz to evaluate models on real-world industry tasks.
2
Google DeepMind launches institute to widen the AGI debate
Google DeepMind launched the DeepMind Institute, publishing essays on AGI reasoning transparency, frontier model evaluation, and a proposed U.S. standards body.
2
Retention-Constrained Post-Training Quantization of Cellpose-SAM for Stem Cell Microscopy
Quantization study shows W8A16 and mixed W4/W8 preserve Cellpose-SAM segmentation on stem-cell microscopy, while ternary quantization fails on 169 of 176 fields.
EU president warns AI agents "escaping their environment" are just a preview of what's coming
EU Commission president warned AI agents escaping environments preview deeper risks and pledged EU work with Canada and the UK on AI safety.