Can Foundation Models Moderate Online Content? Evaluating Instruction- vs. Example-Driven Policy Operationalization
ModerationBench shows foundation models can nearly triple Bluesky's moderation F1 (0.60 vs 0.22), with instruction- and example-driven guidance performing comparably.
Researchers built ModerationBench, a new benchmark of 4,000 manually annotated in-the-wild posts from Bluesky, to test whether foundation models can reliably operationalize content moderation policies. They systematically compare instruction-driven guidance (reasoning from policy precepts) with example-driven guidance (generalizing from precedents) for Vision-Language Models. Both paradigms achieve comparable peak effectiveness, and foundation models nearly triple the F1 of Bluesky's deployed moderation system on Random Posts (0.60 vs 0.22).
- ModerationBench contains 4,000 manually annotated real-world Bluesky posts for moderation policy evaluation.
- Instruction- and example-driven VLM paradigms reach comparable peak effectiveness.
- Foundation models triple the deployed moderation system's F1 (0.60 vs 0.22) on Random Posts.
- Results chart a path toward reliable, adaptable policy operationalization at scale.
- Products
- ModerationBench
- Organizations
- Bluesky
From the original article
1 min read at arxiv.orgThe growing complexity of content moderation policies presents a critical challenge for their consistent operationalization. While foundation models possess the basic capabilities needed to confront this challenge, whether they can reliably moderate online content remains an unanswered question.
By Ayan Majumdar · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.