Jailbreak detectors need low false-positive tests, not AUROC alone
Thresholds set on curated benign prompts often failed to hold false-positive budgets on real-user traffic.
Stories per day · 30 days
8 RSS
Relateddecision-models3abstention1agent-cost1agents1benchmarks1distilbert1distribution-shift1false-positive-rate1gaussian-filtering1guardrails1intern-decision1intrusion-detection1jailbreak1jailbreak-detection1latency1latent-force1llm-agents1llm-evaluation1
48 stories, newest first · one card per story
Stories tagged #calibration
MIRA: Mutual Information guided calibration for Reliable Test-Time Adaptation
MIRA is a source-free online adaptation method using lightweight Gaussian perturbations and mutual information to improve test-time adaptation under distribution shift.
Paper revises latent force ODE filters for stability and calibration
The joint posterior is a reweighted conventional posterior, and state prior variance controls how observations inform the force.
Cost-aware abstention cuts wrong routes on IT ticket classification
On 47,837 tickets, a calibration-selected TF-IDF policy cuts test cost 38.1% and wrong routes from 15.3% to 3.5%.
VX-One study tests typed decision heads for perimeter triage
A 100K–10M-parameter model is measured against XGBoost and a Suricata cascade on public IDS sets.
DecisionPivot benchmark separates decision accuracy from confidence
Two 27B models only 2.6 points apart on the reference track score 70.1% and 44.2% on contrast groups.
Jev 1.13 raises agent cost 19% without improving pass rate
On 21 held-out Terminal-Bench 2.1 tasks, Jev matched a 73.0% pass rate but cost 22% more than a placebo.
Intern-Decision: Efficient Multimodal Decision Making with Autoregressive Masked Language Modeling
Intern-Decision's 4B model hits 90.02% decision accuracy with 44 ms latency on an RTX 4090.
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
ProbabilityBench gives exact class probabilities so researchers can measure model calibration and Bayes-error estimates.
StartLux-Decision: AutoThink Decision Models with Reinforcement Learning for Calibrated Deliberative Decisions
StartLux-Decision models from 0.8B to 27B add confidence-guided reasoning trained with RLCD².
Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection
A learned answer selector on Qwen3-14B improves TriviaQA coverage and lowers scored-answer errors versus confidence filtering.
Where Can a Decision Model Diagnose HVAC Faults? Reasoning Demand, Physical Representation, and Robustness Under Shift
Decision model Jev diagnoses simple HVAC faults from physical features but not context-heavy faults, and stays calibrated under shift.
Learning joint probabilistic weather forecasts from station observations alone
CLARA learns joint probabilistic surface-weather forecasts from station observations without numerical weather models.
A Developer’s Guide to Laya: Zero-Shot Decisions and Calibration
A coding guide shows how to use Laya for zero-shot decisions, calibration, and abstention on CLINC150.
System Switch: When Should a Fast Decision Model Stop and Think?
Researchers test when a fast Doom actor should defer uncertain decisions to a slower reasoning model.
Detecting a Shift Is Not Enough: Exact Minimax Limits of Linear Representation Repair
Exact minimax limits show linear shift repair needs far more calibration signal than detection.
Certification of Real Images through Calibrated Content Authentication
Deepfake detectors fall from 99.5% to 76% accuracy; calibrated authentication bounds false certification at 1%.
SearchJev: A Fast and Calibrated System-1 Model for Search Agents
SearchJev makes calibrated search decisions 5.2× faster than Qwen3.5 and lifts BrowseComp-Plus accuracy from 45% to 54%.
LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them
LLM-as-Jev reads calibrated categorical decisions from token probabilities, matching specialized models on untuned Qwen3.5-4B.
From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders
Researchers turn biomedical sentence encoders into calibrated typed-decision models and release MEDLINE-S1.
ReCIRC: Rectified Conformal Risk Control
ReCIRC reparameterizes conformal risk control so thresholds target conditional risk while keeping finite-sample marginal guarantees.
A Unified Uncertainty Representation for Graph Neural Networks via Doubly-Spectral Stochastic Expansion
DSS-GNN unifies GNN calibration, OOD detection, and shift robustness in one spectral representation.
From Migration to Calibration: Preserving Agent Capabilities across Models, Jurisdictions, and Scale
Researchers propose three-layer calibration so agents keep capabilities when models, jurisdictions, or scale change.
COGNIT-Guard: Calibrated Standalone Direct-Decision Guardrails with Heterogeneous CPU-NPU Confidence Cascading under Explicit Latency and False-Positive Constraints
COGNIT-Guard cascades a CPU gatekeeper to a 322M model for low-latency prompt safety screening.