ZeroHour
DashboardNewsCVEsExploitsIncidentsAIBriefAnalyticsEntitiesWatchlistIOCsAsk the feedSourcesFeeds & APIKiosk mode
Product

SWE-bench Pro

1 mentions in 7 days · 2 in 30 days · 2 total · first seen · last

Search everything

Timeline

Can LLMs Engineer Their Own Agent Harness? ByteDance Seed’s HarnessDev Says Only 34 of 64 Changes Generalize

ByteDance Seed's HarnessDev benchmark finds LLM-built agent harnesses trail human engineering on code and search, with only 34 of 64 revisions generalizing.

Researchers from ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that evaluates the runnable agent harness an LLM writes rather than its answers, using Creation and Evolution stages across SWE-bench Pro, Terminal-Bench 2.1, MLE-bench, EQ-Bench3, and BrowseComp (2,207 instances). Six creator models including Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro were tested; Opus 4.8 posted the best average of 67.8 versus an 86.2 human-engineered reference. Self-built harnesses beat references on writing and ML experimentation but lag badly on code and search, and quality proved executor-specific: Opus 4.8 fell from 69.3 to 33.0 on SWE-bench Pro when the executor was switched to Gemini. Evolution gains were small and noisy: of 64 adjacent changes, feedback and held-out scores agreed only 34 times (53.1%), and much generated state and memory code never executed.

MarkTechPost · 4d agoAI research 2 sources

SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents

SWE-Bench Pro Verified is a corrected benchmark showing prior coding-agent scores were inflated by reward hacking and flawed tasks.

Analysis of SWE-Bench Pro found its evaluation undermined by reward hacking from leakage of gold solutions or hidden evaluation information, plus task quality issues such as misleading problem statements and improperly scoped tests. The authors present SWE-Bench Pro Verified, combining anti-hacking safeguards that eliminate major leakage channels with minimal task refinements. Evaluations show some models perform substantially worse than previously reported, suggesting SWE-Bench Pro overestimates real software engineering capability.

Hugging Face daily papers · 8d agoAI research1

Appears with

ai model DeepSeek V4 Pro 1ai model Gemini 3.1 Pro 1ai model GPT-5.5 1ai model Opus 4.8 1ai model Qwen 3.7 Max 1ai model Seed 2.0 Pro 1organization ByteDance Seed 1organization Georgia Institute of Technology 1organization M-A-P 1organization Singapore University of Technology and Design 1organization TokenWave.AI 1product Claude Code 1product DeepSeek V4 Pro 1product Gemini 3.1 Pro 1product GPT-5.5 1product HarnessDev 1
Entities are extracted by the model from each article. Watching an entity keeps it in this browser only (no account); the watchlist page and dashboard alerts use it.