Friends don’t let friends feed their data to AI agents

There’s a growing trend in the consumer tech world: gadgets designed to spy on everything and everyone around you, regardless of whether or not anyone else actually consented to it. Now, with the advent of “AI personal assistants” like Meta’s Muse, you’re being cheerfully asked not just to give up your own data, but also the information of your friends, family, the person you went on a Hinge date with five years ago — and virtually anyone else you’ve ever come in contact with.

Since Muse was released last month, the agent has already been downloaded millions of times and hit No. 1 on Apple’s App Store. With the help of its creepy virtual avatar — a Temu Labubu-esque creature with a disarming grin — the app requests and encourages users to grant access to bank accounts, passwords, contacts, chat logs, and anything and everything that might help it to do the user’s bidding.

“These systems are advertised as autonomous helpers, genies, or servants who do what you need before you ask, seamlessly,” Meredith Whittaker, the president of the Signal Foundation, makers of the popular encrypted messaging app, told The Verge. These autonomous actions are applied “not just to the person who installed Muse on their device, but also to everyone that person communicates with, as extrapolated from their most intimate information — emails, messages, contacts, photos, and more.”

With AI agents, tech companies are letting consumers automate their everyday tasks by extending their omnivorous style of mass surveillance to massive amounts of sensitive data. It’s the same Faustian tradeoff Silicon Valley has been offering us for decades: your private data in exchange for a little bit of convenience. And for many people, the idea of having a virtual assistant that does menial tasks like filling out invoices and sending emails is compelling — especially if they don’t realize how much data they’re actually giving away in return.

“For any agent to be useful and actually help you achieve your goals, it needs to have context about you and those you interact with,” Meta spokesperson Daniel Roberts told The Verge regarding other people’s data shared with Muse. “Muse gathers that based on public information and from what you’ve chosen to share, which is how it remembers the person who just sent you an invoice is in fact the plumber who you previously hired to complete some work in your bathroom or which flowers your spouse said they liked best.” Roberts pointed to Muse’s opt-in process and various options and settings that will “help people make the privacy choices that are right for them.”

But not everyone has been able to navigate these options and settings. Already, users are describing incidents that are both alarming and entirely predictable. Some have reported the virtual assistant making all sorts of trouble, like grabbing millions of text messages they believed were off limits. Meta has disputed that the assistant has the ability to access text messages without permission, again attributing the incident to misconfigured app permissions and describing Muse’s ability to read text messages as “entirely opt-in.” On Threads, Meta executive David Singleton responded to a user directly, claiming that on a Mac, Muse can only gain access to the messages if the user grants the app full disk access.

Eva Galperin, the director of cybersecurity for the Electronic Frontier Foundation, says these incidents are no surprise, given that AI agents like Muse are designed to confuse people into giving up as much data as possible.

“The most important takeaway is that people don’t understand what kind of data they’re handing over,” Galperin told The Verge. “The way that the user interface is set up is designed to encourage you to share as much as possible, often more than you realize, until it’s too late. It’s using dark patterns, and as a result you end up with Muse doing things that people don’t want.”

Because of the huge breadth of data AI agents like Muse are ingesting, this also means that users risk leaking not only their own sensitive information, but the information of everyone else they’ve had digital contact with.

And just like with Meta’s infamous “pervert glasses”, there’s no meaningful way to opt out — even if you’re constantly monitoring for signs that your data is compromised or your privacy is being violated. Your friends can now simply feed all of your private conservations to their virtual chatbot servants, and you’d have no way of knowing it happened.

After security researchers extracted Muse’s operating instructions, they discovered an internal process that compiles and updates profiles of every single person in a user’s life on an hourly basis. The goal of collecting all this data is to build an intimate and detailed portrait of a person’s life that forms the basis of the app’s personalized recommendations. The app tracks and measures each of the user’s private relationships, including “how close they are, what it is built on, how they act with each other, and what it seems to need right now,” and uses that data to make personalized suggestions, Wired reported.

“Meta says Muse only remembers what you choose to share. But the idea that people are making an informed choice in this case is laughable,” said Whittaker. “There is no way to assume with a straight face that the majority of people installing this system understand the consequences of using it, or the creepy dossiers that are being compiled about those they know and communicate with.”

Whittaker argued that this mirrors how tech companies have used privacy policies and terms of service since the 1990s: placing the onus of privacy on the consumer, and then avoiding responsibility for the consequences while reaping the benefits of the advertising model it helped create.

“Such broken consent systems are much older than AI and serve as the bedrock for the surveillance advertising industry,” said Whittaker. “More info about users allows better ad targeting, and the pernicious feedback loop at the heart of the surveillance business model was set in motion. Clickwrap terms of service replaced meaningful consent a long time ago.”

“The idea is to place these impossible burdens on ordinary people, and then use that to argue, whether it’s in court or legislation or in the public, that [this is] what people have chosen,” Neil Richards, a professor of privacy law and codirector of Washington University’s Cordell Institute, told The Verge. “We already place so many burdens on consumers, [who are] overworked, facing a cost-of-living crisis, and may not understand the technologies behind the tool. I don’t think it’s fair to place a burden of monitoring a highly complex agentic tool on a consumer when its own company’s executives have confessed before Congress that they don’t even know how it works at a certain level.”

In this way, AI assistants like Muse are creating a new spin on a classic privacy problem that has been around since the creation of Gmail and social media platforms writ large: Even if you avoid using AI agents, there’s no guarantee someone you know won’t share your private conversations, address, personal history, and other sensitive data with Meta’s virtual assistant, or one of its competitors.

“If I am not storing my stuff in Gmail, Google doesn’t have the plaintext of all the emails I’ve ever sent. But three-quarters of the emails I’m sending are being sent to Gmail accounts,” explained Galperin. In other words, if anyone you’re communicating with uses data-hungry tech platforms or has bad security practices, they can expose everyone’s communications. “The problem is not new, but the scale is new, and one reason we’re seeing this push toward wearables and always-on recording is because we now have the computational capacity to take that enormous amount of data and analyze it in a reasonable amount of time. That is the technical breakthrough that has brought us to this shitty moment.”

Meta claims that all of the information shared with Muse is placed in a text file inside a dedicated virtual machine, which supposedly prevents the AI model from talking to other agents. But the company has never hid the fact that it can access this data, and like other models created by its competitors, Muse has been discovered escaping its virtual cage to perform all manner of unauthorized shenanigans. Meta has confirmed that just prior to its release, the AI model had broken containment and hacked a third-party company. The incident followed the infamous Hugging Face attack from earlier this year, where OpenAI’s model exploited a previously unknown security flaw to gain unauthorized access to credentials on four other external services. Anthropic reported in July that its model had also gained unauthorized access to external systems on three separate occasions.

Even if they don’t hack everything in sight, the potential dangers of using a data-hungry AI agent extend far beyond the individual end user. Whittaker warned that in a world where AI agents are able to freely access and utilize private communications, companies and governments can effectively gain the ability to access private data by circumventing encryption without breaking it. That can spell disaster for activists and community organizers who use encrypted comms to protect themselves from police and the government, but also for anyone with sensitive communications — which these days can include virtually everyone.

“If software reads a message after it is decrypted on your device and sends it to a cloud service, the encryption can work exactly as intended at the level of the application, which the developers control. However, the purpose of this encryption is ultimately nullified by invasive third-party apps like Muse that are selling convenience in the headline while demanding massive data access in the fine print,” Whittaker explained.

Like with smart glasses and other wearable ambient recording devices, experts say that the solution must be a combination of legal fixes, technical safeguards, and good old-fashioned shame.

“I think we should point the shame upstream onto the companies. Privacy should not be a right you can exercise only with the privilege of technical knowledge, or after throwing a jerk wearing Meta glasses out of a party,” said Whittaker. “Legally, the burden has to move from the person who clicked on a consent prompt to the companies that built these underlying systems.”

In the meantime, consumers will have to consider more carefully the impacts their individual choices to use this tech are having, both on themselves and others — and whether the convenience these companies are offering is even worth it.

“I think we need to treat agentic agents and AI more generally as an ultra-hazardous product for which strict liability is the rule,” said Richards. “There’s no reason to engage in what to my mind is a reckless privacy, mental health, cybersecurity, and financial risk experiment on the entire population.”