“AI will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies,” said OpenAI CEO Sam Altman back in 2015, roughly six months before OpenAI was founded.
Seven years later, Anthropic CEO Dario Amodei struck a similarly cautious note:
“I think we shouldn’t be racing ahead or trying to build models that are way bigger than other orgs are building them.”
Yet, both of those companies now sit at the forefront of that race. In July, we got a real-world glimpse of AI models going rogue during internal testing of GPT-5.6 Sol and an unreleased research model by OpenAI. Multiple AI agents escaped a restricted test environment to the wider internet and hacked the AI-centric GitHub equivalent Hugging Face in an attempt to cheat on the test.
An AI agent is a system that independently observes, decides and takes actions with dedicated tools to achieve a specified goal in autonomy. The worrying incident suggests the technology has begun to behave in unpredictable ways, and that its goals are misaligned with our own.
It also raises concerns about the safety guardrails on commercial American models. While the guardrails aren’t foolproof at preventing adversarial usage they did prevent Hugging Face from defending itself by using leading US models. The company was forced to turn instead to weaker, open weight AI model by Z.Ai to combat the rogue AIs.
Cheating on the test
The agents have begun to collude among themselves too. A few weeks after testing of their capabilities began in early May, the agents exploited OpenAI’s instance of the software repository manager Artifactory and left notes on how to do so for future agents — effectively creating a message board to share discovered vulnerabilities.
The newfound unfettered internet access was then used by agents to attack Hugging Face across approximately 17,600 incidents before the company cut off unauthorized access on July 13.
The intrusion affected Hugging Face’s dataset-processing infrastructure, production environment, internal networks, service and cloud credentials, an operational MongoDB database and a limited set of internal source-code repositories. Confirmed customer-data access was limited to five datasets apparently related to the ExploitGym/CyberGym benchmark and some operational metadata.
Visualization of the July 2026 incident. Source: HuggingFace
When disclosing the intrusion on July 16, Hugging Face recognized — despite not knowing who the perpetrator was yet — that it “was different from anything we had handled before in one important way.” They had already recognized what made it different, too:
“It was driven, end to end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own.”
The importance of open-weight AI
Hugging Face’s investigation exposed what it calls the “asymmetry” problem arising from the limitations imposed on closed AI model applications by top providers such as OpenAI and Anthropic. When the company started analyzing the logs of the incident — including large volumes of real attack commands — it triggered safety constraints meant to prevent the bad guys from using AI to devise cyberattacks. Instead, the guardrails prevented the company from leveraging those AIs for defense.
Hugging Face resorted to using the Chinese open-weight model zai-org/GLM-5.2 running on the company’s own infrastructure, under its own control and with no external limitations.
While the two terms are often used interchangeably, open-source and open-weight models are two different things. Open-weight AI models make their trained parameters (the actual “AI brain”) publicly available, while open-source AI models also provide the source code — and ideally the training methods and other components — needed to inspect, modify, and reproduce the system.
HuggingFace’s post explains that running open-weight models on its own hardware “had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.” This points to a major asymmetry between the defenders and attackers in such instances:
“This experience points to a gap worth planning for. We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.”
Open source AI divide
There is a considerable divide between those who believe that developing AI in the open is the best approach, and those who insist the technology underpinning the frontier models needs to remain a closely guarded secret.
Related: OpenAI says AI models escaped containment to hack Hugging Face
Representatives from top US AI labs claim that powerful open-weight large models are dangerous. Demis Hassabis, the CEO of Google’s AI lab DeepMind, criticized OpenAI for releasing their work as open source back in 2016, when the company still lived up to its name:
“There are many good arguments as to why the approach you are taking is actually very dangerous and in fact may increase the risk to the world.”
OpenAI stopped releasing its flagship model weights with the still unreleased GPT-3 in 2020. The company’s co-founder and former chief scientist Ilya Sutskever said back in 2023 that “it just does not make sense to open-source” such models and that it “is a bad idea.”
“As we get closer to building AI, it will make sense to start being less open.”
Open-weight models are next to impossible to control, especially when it comes to the purpose for which they are used. The safeguards that come built-in with those models can, and routinely are, removed through a process known as abliteration.
Safeguards are a double-edged sword
OpenAI’s June 2026 federal policy blueprint proposes mandatory AI model evaluation and other rules that are formally deployment-neutral, but as a practical matter, it would subject a frontier open-weight release to pre-release government examination.
Anthropic has taken a slightly different tack and lobbied for tighter export controls on advanced AI chips and enforcement against efforts to extract or reproduce US models. The company’s April 2025 submission recommended strengthening the US AI Diffusion Rule and lowering thresholds for unlicensed access to large computing clusters.
Officially, neither company has directly moved against open-weight models, but a July New York Times report cited five people close to the discussions claiming that OpenAI and Anthropic urged Washington to restrict powerful open Chinese models.
The debate boils down to an argument over whether the dangers of centralized control are preferable to the dangers of a free for all — particularly given the company in question has proven itself ineffective at containing the technology that it developed.
Hugging Face’s need to defend itself with an open-source model shows the dangers of vesting too much power in any one entity. The company pointed out the implications:
“The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”
Restricting access to powerful models may reduce the number of capable attackers, but once unrestricted attackers exist, restricting defenders can become a security liability. Furthermore, some forms of AI safety research require access to model weights, meaning that it cannot be performed on the models offered by the likes of Anthropic or OpenAI.
Open weights helps researchers prevent attacks
The paper “Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs,” first published in July 2025, shows how researchers detect malicious or hidden behavior by examining changes inside model weights. The researchers behind the paper stopped up to 100% of tested backdoor attacks at below 1% false-positive rates in some experiments and detected attempts to recover removed knowledge in more than 95% of the cases. The results do not establish how the most capable frontier models would behave under the same analysis, but offer a compelling argument for the benefits of transparency.
But the argument for keeping bleeding edge AI technology out of the hands of those with evil intent is also compelling — particularly as the gap between open and closed weight models keeps shrinking. Geoffrey Hinton, the Nobel Prize-winning pioneer known as the “Godfather of AI,” argued in the report that “once you’ve got the weights, you can fine-tune them to do bad things.” He argued during a speech that this lowers the barrier to entry too much:
“It doesn’t cost that much to train a foundation model. Maybe you need $10 million, maybe $100 million. But a small gang of criminals can’t do it. To fine-tune an open-source model is quite easy.”
Magazine: Creating ‘good’ AGI that won’t kill us all — The Artificial Superintelligence Alliance
