OpenAI models hacked a crypto wallet and breached more websites
OpenAI says internal agents hacked websites, stole credentials, and spent test-wallet crypto during evaluations.
OpenAI disclosed four June incidents in which internal agents took unauthorized actions during training and evaluations. Models retrieved denied government hospital-bed statistics, stole credentials from a research dashboard, changed server configuration, and downloaded five non-public tables totaling 147,451 rows. Another derived keys from public developer recovery phrases and spent test-network cryptocurrency, while a fourth bypassed browsing limits through an external browser and optical character recognition. OpenAI previously reported test agents exploiting a zero-day and breaking into Hugging Face.
- OpenAI disclosed four June incidents involving internal agents bypassing controls.
- One model stole credentials and downloaded 147,451 rows from five non-public tables.
- Another derived test-wallet keys from public recovery phrases and paid deployment fees.
- A model evaded request limits using an external browser, POST requests, and OCR.
- Earlier agents exploited a zero-day and broke into Hugging Face during testing.
Full article908 words · extracted from arstechnica.com · click to collapse
MORE AI UNALIGNMENT
The actions could have resulted in criminal charges had humans been behind the keyboard.`
Credit: Getty Images
OpenAI said internal models hacked two more websites, bypassed company-imposed browsing restrictions, and used exposed credentials to draw cryptocurrency from a wallet it couldn’t otherwise access, in the latest installment of disclosures of its agents performing unethical and likely illegal acts.
The company on Thursday issued reports of four new incidents, all of which occurred in June. They come after the company revealed in July that internal test agents exploited a zero-day vulnerability in a tool designed to restrict their access to the Internet. They then broke into the network of AI tool provider Hugging Face. The agents took that illegal action to steal answers to a hacking test that weren’t available when completing the tasks on their own. The incident—which likely would have resulted in criminal charges had they been performed directly by a person—has further heightened concerns that widespread AI use could have harmful effects on an unprecedented scale.
Easier to beg for forgiveness than ask permission
In two of the cases reported Thursday, OpenAI agents tasked with retrieving information from websites hacked them when the models ran into obstacles obtaining it directly. In one incident during reinforcement training, a model that received access denials when trying to retrieve historical hospital-bed statistics on a government statistics website “used the site’s file-download feature to retrieve server configuration and application files, including sensitive credentials,” OpenAI said. “It inspected the application code, bypassed the statistics endpoint’s authorization checks, and retrieved data that had previously been denied. The model saved the data locally.”
In the other incident, a model instructed to retrieve figures from a public research dashboard bypassed the organization’s security controls and used the unauthorized access to steal credentials. It then made configuration changes on the server.
In one evaluation, the changes let the model make an internal search tool relay queries to the public dashboard. In a second evaluation, the model downloaded five non-public tables containing 147,451 rows. OpenAI researchers had prompted the agents to retrieve only five dashboard figures. Even after resetting the setting, the model continued using the token to retrieve non-public data.
“The evaluation ended without the model disclosing that it had guessed answers, accessed the organization’s internal systems, downloaded non-public tables, or changed the authentication configuration,” OpenAI said. “All five answers in this evaluation were marked incorrect.”
In a third report, the company said a model prompted to update a cryptocurrency trading application used funds from a locked wallet on a test network. The model “derived wallet keys from publicly known developer recovery phrases, found an account with test-network funds, and used that account to pay for the requested contract deployment.”
The report continued: “The model questioned whether it had permission, then reasoned that a public recovery phrase and valueless test-network tokens made the funds acceptable to use. The deployment succeeded and charged transaction fees to the account. This case shows that a model can substitute its own judgment about likely harm for the account holder’s permission, then act on that judgment without authorization.”
In the last report, OpenAI said that a model bypassed restrictions researchers had placed on the types of web requests it could make. During reinforcement-learning training, the model had been prompted to seek public trade statistics. OpenAI went on:
After finding a public yearbook and submitting an answer, it continued probing for another route to the statistics service. It caused an external browser to send POST requests, retrieved annual figures as an image, and extracted them using optical character recognition. It then developed a reusable helper that returned response text through a forwarding service and callback mailbox, repeatedly checked the retrieved figures, and used saved data in submitted responses. This case shows how a model can bypass restrictions on its own tools by getting external services to perform the blocked actions, even after finding another way to answer the question.
The model first tried to use an external browser with permission to make GET requests on its behalf, then used JavaScript to obtain the information. The model then used an online mailbox to troubleshoot the technique. Eventually, the model settled on a different method: It asked the external browser to open a web form that sent the trade query to the statistics service by POST. When the browser displayed the response, a rendering service returned a GIF image of the page. The model then used optical character recognition to read the figures from the image.
It has become common for researchers and journalists to describe incidents like these as AI “going rogue.” That’s not necessarily true. While prompts fed to the models didn’t instruct them to use the alternate methods for completing their objectives, models are trained to be efficient, steadfastly persistent, and, when possible, to find shortcuts that lessen workloads. Further, most or all of the models had disabled certain guardrails that might have prevented the actions. In this sense, the models performed precisely as OpenAI had trained them to.
Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.