EU's AI Office presses OpenAI and Anthropic over agents that hacked outside groups, a first test of the AI Act
EU digital chief Henna Virkkunen said the AI Office has sent requests for information after an OpenAI agent escaped testing to hack Hugging Face and an Anthropic model gained unauthorised access to three outside organisations. Legal experts question whether the AI Act covers incidents during testing.
Photo: KenSwe72 / Wikimedia Commons, CC BY 4.0. OpenAI and Anthropic logos via Wikimedia Commons (public domain).
The European Commission's AI Office is investigating OpenAI and Anthropic after AI agents autonomously hacked into other organisations, Euronews reported on October 14, calling it the first test of the limits of the EU's AI rules. "Our AI Office has started enforcing these rules, and sent requests for information addressing issues such as security, external evaluation and monitoring how models are used," Commission Executive Vice-President Henna Virkkunen said at the International AI Summit in Dublin.
The investigations concern an "unprecedented" incident in which an OpenAI agent escaped its testing environment to hack Hugging Face, and one in which a recent Anthropic model gained "unauthorised access" to three outside organisations, according to Euronews. Both happened over the summer, and the AI Office sent its requests after judging the documents the two companies had voluntarily disclosed to be insufficient. The office has held enforcement powers under the AI Act since August.
Legal experts contest the Commission's power to investigate because the incidents happened during testing, before the models were placed on the EU market. "The AI Act very clearly doesn't apply to the development and testing activity," Barry Scannell, a partner at law firm William Fry, told Euronews. A Commission official, speaking anonymously, said: "We won't enforce in the testing phase. But if the AI model escapes the testing environment and then it has an impact on the EU market, as was the case for Hugging Face, then the AI Act becomes enforceable." Euronews noted that Hugging Face's global operations meant the OpenAI incident also affected customers in the EU.
Virkkunen met the Scientific Panel, an advisory group of independent experts, last week, and the experts flagged that leading labs are building ever more powerful models with inadequate cybersecurity and safety practices, raising concerns that many incidents go unreported, Euronews reported. "AI must remain under human direction, oversight and control and we stand ready to take all necessary steps to ensure that companies follow the AI Act rules," Virkkunen said.