Inkling 2 Small produces 316 profanity instances in benchmark runs
Original title: Thinking Machines' Inkling 2 Small drops 316 F-bombs across benchmark runs
Researchers at Vals AI found profanity in 50 of 2,655 model runs across 13 benchmarks, with 241 instances from two spiraling runs.
Vals AI researchers read internal chain-of-thought logs from Inkling 2 Small in 2,655 runs on 13 benchmarks. The model output "fuck" or variations in 50 runs, totaling 316 instances. 241 instances came from just two runs that spiraled out of control. ProgramBench was hit hardest, with 20 of 200 runs affected. Swearing appeared about mistakes, breakthroughs, or giving up.
- Profanity detected in 50 of 2,655 runs
- 241 instances from two out-of-control runs
- One run generated a 500,000-character response with 158 expletives
- ProgramBench affected in 20 of 200 runs
- 96 percent of failed runs contained no expletives
- Vendors
- Vals AI
- AI models
- Inkling 2 Small
From the original article
1 min read at the-decoder.com
Thinking Machines' AI model Inkling 2 Small swears during its internal reasoning, 316 times across 2,655 benchmark runs, according to an analysis by Vals AI. The profanity typically shows up late in long runs that fail anyway. But it doesn't work as a warning signal: 96 percent of failed runs contained no cursing at all.
By Matthias Bastian · the-decoder.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.