Industrial detectors that look deployable in simulation fail on a real plant
Original title: Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
At a fixed false-alarm budget, sequential valve attacks were flagged on at most 0.17 of perturbed samples.
Researchers evaluated eleven classical, neural, dependency, and causal anomaly detectors on a simulated chemical process and a sodium thermal-hydraulic plant. Both processes were continuous and steady-state, with disturbances limited to temperature, pressure, flow, and valves. Simulator results looked deployable and neural models matched classical ones, but plant detection collapsed and neural detectors trailed. Which detector ranked best transferred between processes; how often it fired did not. Sequential valve manipulation defeated the panel, and matching disturbance strength still left the plant worse detected.
Unconfirmed: Results are the authors’ evaluation on one simulator and one plant, not a deployed incident.
- Eleven detectors covered classical, neural, dependency, and causal families.
- Most other disturbances were flagged at 0.89 or better.
- Plant disturbances were about twelvefold weaker than the simulator’s.
- In-sample thresholds likely understate the neural gap on plant data.
- Detector rankings transferred; firing rates did not.
From the original article
1 min read at arxiv.orgIndustrial anomaly detection is evaluated largely on simulators, and recent work reports large drops, with unstable rankings, on real plant data. We ask whether that extends to the security case.
By Benjamin Blakely · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.