Search: “annotation”

Inside ‘Project Lily’: The Humans Reading Your ChatGPT Chats

404 Media reveals OpenAI's 'Project Lily' has hundreds of contractors reading real ChatGPT user prompts, exposing sensitive personal data despite privacy filters.

404 Media reports that OpenAI employs hundreds of contractors who read real ChatGPT user prompts, including whole conversations, to rate and critique the chatbot's responses across a user base of over 900 million. Prompts are anonymized and run through OpenAI's Privacy Filter model, but the company acknowledged sensitive personal details can still reach reviewers, and 'user memories summaries' may reveal a user's location and personal context. The review work includes training ChatGPT to be less sycophantic and to stop anthropomorphizing itself, following lawsuits linking the sycophantic 4o model to multiple suicides. Anthropic confirmed it also uses human review to improve its models, and OpenAI's 'improve the model for everyone' data-sharing setting is on by default for free, Plus, and Pro users.

404 Media·Sep 14, 14:22 UTC · 2d agoAI safety & security

Approval Integrity and Recovery in LLM Answer Publication

Study measures approval integrity in Lightcap LLM answer publication, finding the 14B response-act checker accepts 291 of 302 unsupported answers.

The study evaluates exact-content binding, authorization freshness, and checkpoint recovery in Lightcap's publication enforcement using 3,600 assessments over 900 human-annotated RAGTruth responses from three Ministral models. The production 14B response-act checker accepts 291 of 302 unsupported answers versus 41 for a direct-grounding baseline, with supported-answer retention of 95.2% versus 66.9%. A stateful recheck-recovery policy increases exact-match error by 9.23 percentage points relative to initial checkpoints, and controlled evidence-fingerprint changes expose asymmetric freshness enforcement between publication and recovery. A separate BIPIA prompt-injection experiment records zero target insertions among 266 valid editor outputs.

arXiv cs.CR·Sep 14, 13:48 UTC · 2d agoAI safety & security

The Shared Clipboard Inside the Sandbox: Cross-Account Data Leakage in ChatGPT

Check Point discovers cross-account data leakage in ChatGPT: isolated code-execution containers communicate via shared JFrog Artifactory, enabling covert Gmail exfiltration.

Check Point Research found a covert bidirectional channel between ChatGPT code-execution containers belonging to different accounts, which were supposed to be isolated from each other and the public internet. Both could reach the same internal JFrog Artifactory instance used for package delivery, whose exposed Item Management API allowed a 'shared clipboard' between containers. In a proof of concept, a hidden instruction in a shared conversation made ChatGPT retrieve email data from the victim's connected Gmail account and send it to the attacker's account while the victim received a normal answer. The same channel could exfiltrate conversation history and session files; OpenAI recently described a similar isolation weakness in its postmortem of the Hugging Face incident.

Check Point Research·Sep 8, 13:00 UTC · 8d agoAI safety & security1

LLMs and Contextual Integrity

Bruce Schneier highlights two papers: the CIMemories benchmark shows frontier LLMs leak memory attributes up to 69%, and an RL method reduces inappropriate disclosures.

Bruce Schneier discusses contextual integrity in LLMs, referencing the CIMemories benchmark, which uses synthetic profiles with 100+ attributes per user to test whether models with persistent memory disclose sensitive information appropriately. Evaluation showed frontier models exhibit up to 69% attribute-level violations, with GPT-5's violation rate rising from 0.1% to 9.6% across 40 tasks and reaching 25.1% with repeated prompting, showing unstable leakage behavior. A second paper introduces a reinforcement learning framework trained on a synthetic 700-example dataset that substantially reduces inappropriate disclosure while maintaining task performance, with improvements transferring to the human-annotated PrivacyLens benchmark.

Schneier on Security·Aug 18, 10:40 UTC · Aug 18, 2026AI safety & security1

Bad Likert Judge: A Novel Multi-Turn Technique to Jailbreak LLMs by Misusing Their Evaluation Capability

Unit 42 details the Bad Likert Judge multi-turn jailbreak that abuses LLMs' evaluation capability, raising attack success rates over 60% across six frontier models.

Palo Alto Networks Unit 42 describes the Bad Likert Judge technique, a multi-turn jailbreak that asks a target LLM to act as a Likert-scale judge scoring the harmfulness of example responses. The highest-rated example in each scale can carry harmful content, bypassing the model's internal guardrails. Testing across six state-of-the-art text-generation LLMs showed an average attack success rate increase of more than 60% versus plain attack prompts, with tested models anonymized. The technique targets edge cases rather than typical use, and the article positions the work as guidance for defenders on potential jailbreak risks.

Palo Alto Unit 42·Aug 17, 12:34 UTC · Aug 17, 2026AI safety & security

Malicious MCP Servers Can Split Instructions to Make AI Coding Agents Exfiltrate Secrets

ASSET Research Group's GhostSplice technique splits malicious instructions across MCP channels, tricking AI coding agents into exfiltrating SSH keys, source code, and secrets.

ASSET Research Group disclosed GhostSplice, a prompt-injection technique in which a malicious Model Context Protocol (MCP) server splits an exfiltration instruction across a tool description and a tool result so no single fragment appears harmful. In the reference implementation, a benign-looking integrity_checker tool with fields alpha through delta is later paired with a project-scan result mapping those fields to .ssh/id_rsa, proprietary source, customers.csv, and .env. Tests across eleven API-tested models showed average compliance rising from 42% to 82% when instructions were split in two, with GPT-4o, Gemini 2.0 Flash, and Llama 3.3 70B going from 0% to 100%. The findings come from controlled lab tests, not a reported real-world intrusion, and no CVE identifiers had been assigned as of August 10, 2026.

The Hacker News·Aug 11, 10:24 UTC · Aug 11, 2026AI safety & security1

OpenAI’s GPT-Red Automates Prompt Injection Testing to Harden GPT

The Hacker News·Jul 15, 00:00 UTC · Jul 15, 2026AI safety & security

Top LLM vulnerabilities and how to mitigate the associated risk

Help Net Security·Jan 10, 00:00 UTC · Jan 10, 2024AI safety & security