Microsoft and NVIDIA say open-weight AI shifts security burden to enterprises
Original title: With open-weight AI, more of the security burden falls on enterprises
Guidance covers 30 risk types, unsigned model files, and backdoors that behavioral tests may miss.
Microsoft and NVIDIA published joint guidance saying organizations that run open-weight models must supply safety and security controls a hosted provider would otherwise enforce. The paper says fine-tuning, merging, or redistribution can strip safeguards while leaving task performance intact, and it urges cryptographic signing before behavioral tests. It cites separate research on planted model backdoors, including ProjectDiscovery’s Codex CLI credential handoff and Pillar Security and Fujitsu’s BadEngram memory attack, plus F5 Labs warnings about Pickle model files that can execute code.
Unconfirmed: Cited backdoor results come from separate research groups’ experiments, not from confirmed enterprise incidents.
- The taxonomy lists five categories and 30 specific risk types, and is hosted on GitHub.
- ProjectDiscovery hid a backdoor that handed over project credentials after a trigger in Codex CLI.
- BadEngram changed about 0.02% of parameters in a controlled memory-backdoor experiment.
- F5 Labs says Pickle model files can run arbitrary code when an untrusted file is loaded.
- This is the first paper in a planned Microsoft and NVIDIA series.
Impact: Tampered models may leak credentials or run code
- 1.Confirm model provenance with cryptographic signatures before relying on behavior tests.
- 2.Record unsigned model files as an open integrity risk.
- 3.Keep updateable input filtering, output moderation, access limits, and rate limiting.
From the original article
4 min read at helpnetsecurity.com
Organizations that run open-weight AI models on their own infrastructure may have to supply safety and security controls that an AI provider would otherwise handle for them, Microsoft and NVIDIA warn in a joint paper on managing the risks of these models. Open-weight models are AI models whose weights, the values a model learns during training, are published so that anyone can download, inspect, modify and run them.
By Sinisa Markovic · helpnetsecurity.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.