Show HN: Parsimony – an open-source benchmark for how much code AI models add
Parsimony ranks AI coding models by net syntax units added on SWE-bench Verified patches.
Parsimony is an open-source benchmark that measures how much code AI models add rather than only solve rate or cost. It parses published SWE-bench Verified patches, counts syntax units such as statements, calls, comparisons, names, and literals, and subtracts deleted code. Only Python is measured, tests are not rerun, and models are ranked by mean net units on solved tasks. The authors note a moderate correlation between writing more code and solving more tasks, and that all models used mini-SWE-agent.
- Parsimony counts net syntax units added, not lines of code.
- Rankings use published SWE-bench Verified results without rerunning tests.
- Only Python changes are parsed; tests, docs, and generated files are excluded.
- More code correlated with higher solve rates, Spearman 0.33 to 0.60.
- Every measured model used the same mini-SWE-agent.
Full article315 words · extracted from parsimony.lasu.dev · click to collapse
Existing benchmarks measure solve rate and cost, but don't punish excess code, which makes a codebase harder to work with in the long term. Parsimony measures how much code AI models add.
It takes the patches that benchmarks already publish and counts coding units: syntax elements like statements, calls, comparisons, names and literals. Lines aren't counted, so cramming code isn't rewarded, and deleted code is subtracted: net units = added − removed.
Compare models
Chart
X axis Y axis
How to read this
Models are ranked by mean net units added on the tasks they solved, lowest first. "Net units added (mean)" also includes failed attempts. Solved comes from SWE-bench Verified; tests aren't rerun. Measured is how many solved attempts could be measured.
Leaderboard
Task| Rank | Model |
|---|
Missing a model? Request it. A number looks wrong? Report it.
Method
- Each patch is applied to the task's code, and both versions of every changed file are parsed. Nothing is run.
- Tests, docs and generated files are left out. Only Python is measured.
- Pass/fail comes from SWE-bench's published results; tests aren't rerun.
Limits
- Smaller isn't always better: units measure size, not readability or design.
- Models that write more code also tend to solve more tasks (Spearman 0.33–0.60 on the boards with 26+ models).
- Every model uses the same agent (mini-SWE-agent), but the agent still affects how much code gets written.
Add your own benchmark
Have code-size numbers from another benchmark? Download the example file, fill in your results and open it to get the same chart and table. The file stays in your browser. Format and how to share it.
Open benchmark JSONExample: check for a negative number
Both return the same result.
def has_negative(values):
found = False
for value in values:
if value < 0:
found = True
return founddef has_negative(values):
return any(value < 0 for value in values)