Pilot evaluates Gemini 2.5 Flash Lite without sharing prompts or weights
Original title: Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing
PySyft and an NVIDIA H100 enclave on Google Cloud ran private AILuminate prompts against the model.
Researchers describe double-blind AI safety evaluations that keep evaluator prompts from model owners and keep proprietary weights and inference code on the owner's infrastructure. Secure enclaves encrypt data in RAM at the hardware level. In a pilot, Gemini 2.5 Flash Lite was tested against a selection of the private AILuminate corpus using an NVIDIA H100 enclave on Google Cloud Platform and OpenMined's PySyft. The authors present this as mutual secrecy between one organization's frontier weights and another's closed benchmark.
Unconfirmed: The authors believe this is a world-first use; that priority is not independently confirmed.
- Evaluators want prompts kept from model owners to limit benchmark leakage.
- Model owners keep weights and inference code inside their own infrastructure.
- The pilot used a selection of the private AILuminate benchmark corpus.
- OpenMined's PySyft was used to maintain data and model privacy.
- Products
- PySyftNVIDIA H100AILuminate
- Organizations
- Google DeepMindOpenMined
- AI models
- Gemini 2.5 Flash Lite
From the original article
1 min read at arxiv.orgAI safety evaluations often involve sensitive prompts and proprietary models. Evaluators want to keep their evaluation prompts secret from model owners to prevent purposeful or incidental data leakage, ensuring longer usefulness of their benchmark. Model owners in turn wish to keep their proprietary model weights and inference code within their own secure infrastructure to protect their IP.
By Andrew Trask · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.