MasterControl Seventeen Every Time
Governed enterprise analytics study shows deterministic policy execution matched 110/110 answer-and-evidence contracts while runtime agent planning matched none.
The paper studies a governed approach where a language model interprets the question while deterministic policy selects and runs a pre-approved analytical program returning results and evidence. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B only interpreted intent and policy executed the approved program. None of 330 runtime-planning episodes satisfied the full answer-and-evidence contract, whereas the policy-executed analyzer matched 110 of 110. The authors note this is configuration-specific and expressiveness is preserved via relational operations, aggregation, comparison, windows, ranking, and similarity with replayable results.
- LLM interprets intent while deterministic policy executes pre-approved analytical programs
- 440-run evaluation across three 8B models and Qwen3-8B intent interpretation
- Policy execution matched 110/110 contracts; 0 of 330 runtime-planning episodes matched
- Restricted analytics remain expressive via relational ops, aggregation, windows, and ranking
Full article124 words · extracted from huggingface.co · click to collapse
We study a governed approach to enterprise analytics: a language model interprets the question, while deterministic policy selects and runs a pre-approved analytical program that returns both results and evidence. We show that this restriction can remain expressive within a defined analytical class, using relational operations plus aggregation, comparison, windows, ranking, and similarity. Fixed meaning, policy, data, and execution rules also make results replayable. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B interpreted intent only and policy executed the approved program. None of 330 runtime-planning episodes matched the full answer-and-evidence contract across all test datasets; the policy-executed analyzer matched 110 of 110. This is a configuration-specific result, not evidence that runtime agents cannot succeed under other designs.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.03209