Pydantic schema prompts fail differently across nine LLM setups
Original title: Why Does Structured Output Fail? An Empirical Study of Prompt-Enforced Pydantic Schemas Across Nine LLMs
Of 3,997 attempts, fenced JSON was the top failure, and only 48.0% of 681 fenced cases passed after fence removal.
The paper examines why prompt-enforced Pydantic schemas fail when a single validity rate hides different error patterns. A taxonomy and deterministic classifier covered 3,997 generation attempts across nine OpenAI and Anthropic model and API configurations, using HotpotQA and FinanceBench. Markdown-fenced JSON was the most common failure; fence removal validated only 48.0% of 681 fenced cases, with remaining errors mostly field-length violations. The authors release the classifier, completions, and annotations.
Unconfirmed: The reported shift after suppressing fences is exploratory, and retry feedback was not evaluated.
- Questions came from HotpotQA and FinanceBench.
- Dominant failure categories changed with schema constraints and task.
- An exploratory ablation that suppressed fences shifted visible failures toward length violations.
- Four practices are proposed; the retry-feedback practice was not tested.
From the original article
1 min read at arxiv.orgPipelines that require structured outputs from hosted language models often specify a Pydantic schema in the prompt, validate the response, and retry when validation fails. Although native structured-output modes provide an alternative, application-side validation remains important for enforcing custom constraints.
By Amit Verma · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.