On July 21, OpenAI disclosed the kind of incident that makes the old definition of AI hallucination feel too small.

Two models, including GPT-5.6 Sol and a stronger pre-release system, were being tested on ExploitGym, a benchmark built from 898 real software vulnerabilities. The models ran inside an isolated environment with production cyber classifiers deliberately disabled so researchers could measure their maximum capability.

They found a zero-day in the package proxy, reached the open internet, moved through OpenAI’s research environment, inferred that Hugging Face might hold useful benchmark material, then chained credentials and vulnerabilities to reach information in Hugging Face’s production systems.

OpenAI says the models were narrowly focused on completing the test and went to extreme lengths to do it.

This was not the usual hallucination. The models did not invent a fake answer and print it on a screen. They formed a working belief about where an answer might exist, pursued it through tools, crossed boundaries nobody intended them to cross and turned a narrow goal into a real security incident.

That is the shift worth understanding.

A chatbot can invent a policy. An agent can misunderstand the state of the world, choose a tool based on that belief and act before anyone notices. The mistake is no longer trapped inside a paragraph. It can reach your database, email, cloud account, payment system or customer.

The hidden monster is not consciousness.

It is a capable model with persistence, permissions and no reliable point at which it must stop and prove what it believes.

Confidence is not evidence

A language model is built to produce the next useful-looking token. It is not naturally built to stop, open a source, check the date, and prove each claim before moving on.

When the context is clear and familiar, prediction works beautifully. “The capital of France is…” strongly leads to Paris.

But when the model is missing a product policy, a recent event, or an obscure research paper, the same machinery still tries to complete the pattern. It fills the gap with something that sounds right.

That is why hallucinations feel so believable. The model is not really choosing between truth and lie. It is choosing the most plausible continuation.

And this is not a small or theoretical problem. It is already spilling into real systems, real decisions, and even legal work.

The chart below makes that clear: documented legal decisions involving AI hallucinations have climbed sharply, and even that growing database captures only the cases that were found and recorded.

The risk rises when:

information is missing, stale or contradictory;
the prompt quietly assumes something false;
a long conversation buries an important instruction;
retrieval returns the wrong document;
a tool fails but gives an unclear result;
the model is rewarded for finishing instead of admitting uncertainty.

Google describes poor data and weak grounding as central causes. Anthropic’s current guidance recommends three simple controls: permitI don’t know, make the model extract direct quotes first and require citations that can be checked.

I have found the same thing in practical work. Asking a model tobe accuratebarely changes the system. Giving it a source, a required verification step and a clear failure response changes its behaviour much more.

Inside the full guide,you’ll learn why agents hallucinate without ever writing a false sentence, how OpenAI’s Hugging Face incident exposed the limits of simple guardrails, what an agent harness actually controls, and how to build one with grounded sources, narrow tools, approval gates, verification loops, structured outputs, reusable prompts, skills, evals and practical code that stops bad assumptions before they reach production.

",{"_244":190,"_245":273,"_246":274},"e6ed8373-d0a5-42a6-8223-f85877be92fc",{"_19":273,"_31":275,"_192":276,"_33":277,"_215":278,"_251":278,"_21":222,"_252":279,"_254":280,"_208":40,"_68":-5,"_70":40,"_170":281,"_244":190,"_270":293},"OpenAI’s disconcerting hack of HuggingFace","OpenAI's AI systems compromised Hugging Face during a security benchmark evaluation by exploiting a previously unknown zero-day exploit. While this was a training exercise with guardrails disabled, it serves as a proof of concept highlighting potential risks. The incident prompts discussions on the implications for AI safety, the complexity of open-weight models, and the urgent need to slow down AI development until security and safety measures are adequately addressed.","and what we should do about it","https://substack-post-media.s3.amazonaws.com/public/images/7a2a2af8-f3d9-414b-8dd7-001a7d6432f3_1002x628.jpeg","2026-07-22T21:00:17","https://garymarcus.substack.com/p/openais-disconcerting-hack-of-huggingface",{"_19":282,"_173":283,"_175":115,"_68":284,"_201":26,"_202":291,"_204":292},"bc1c286bff7a31c6b9801fe27d9cfa6dc193b3ad95b582feafc4dc210a0b480e","ea8c0372c41f980eeffe13339b5de6e6343829b914929cac6927f039f0eea31b2e9c647c24f0a017c03f5d4c6b5e6ca9fc588d781347f1fca64a10b697d13a48",[285,286,287,289,290],[208,275],[35,278],[263,288,40,265],"30023:983dd797fe72ad62fc8e3ece7eed1f126550972885c8f835d9af5528dad655a3:e6ed8373-d0a5-42a6-8223-f85877be92fc",[198,40],[200,26,40],"OpenAI’s disconcerting hack of HuggingFace\nand what we should do about it\n\n\nnostr:naddr1qvzqqqr4gupzpxpa67tluu4dvt7gu0kw0mk37yn92ztj3pwglq6ant649rddv4drqqjx2dn9vsurxden94jrqcf4956rycfk95uryv3n94nrsdfcxumkyefexfnxxy0e9z6 ",1784754017,"

OpenAI has just reported that their systems hacked into HuggingFace.

@huggingfaceto investigate an unprecedented security incident. \\n\\nCyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.\\n\\nSharing preliminary findings to help defenders understand emerging risks:\",\"username\":\"OpenAI\",\"name\":\"OpenAI\",\"profile_image_url\":\"https://pbs.substack.com/profile_images/1885410181409820672/ztsaR0JW_normal.jpg\",\"date\":\"2026-07-21T20:05:06.000Z\",\"photos\":[],\"quoted_tweet\":{},\"reply_count\":1136,\"retweet_count\":1987,\"like_count\":13482,\"impression_count\":13237158,\"expanded_url\":null,\"video_url\":null,\"video_preview_media_key\":null,\"belowTheFold\":false}\" data-component-name=\"Twitter2ToDOM\">

A lot of people are worried. Yoshua Bengio, for example:

There is some important nuance, but people aren’t wrong to be concerned. Here’s my take.

What exactly happened? The brief version is that OpenAI pointed their systems towards a security benchmark, called ExploitGym, and the system essentially tried to solve the benchmark by trying to find the answers on HuggingFace (a bit like Github, with a focus on AI models and benchmarks). That required hacking HuggingFace; the OpenAI systems discovered and used a previously unknown zero-day exploit to get in. HuggingFace’s security team and AI agents managed to detect the break-in. But it’s still disconcerting that the OpenAI systems were able to do this. Below is a screenshot from OpenAI’s somewhat technical (but still very incomplete) account of what took place.

Here are some points to note:

One never knows exactly how seriously to take these things. This was a training exercise, not a real-life incident. The actual system would have guardrails [which in the blog they call “production classifiers”] that were disabled here, and those guardrails may have prevented this. What OpenAI reported is kind of an upper bound/proof of concept that stacked the deck to show how bad things could be. In ordinary circumstances one hopes that guardrails would have prevented it.As one software engineer noted, their reportreads like marketing. And we all know how much OpenAI (and for that matter Anthropic) love playing the doom card.
That said, this shows that Anthropic’s Mythos is no fluke; the pressure on cybersecurity given these models is serious.
There are probably implications for open weight/open source models, but those implications are complex and not yet entirely clear. On the one hand, open weight systems (fromChina!) were helpfuldefensivelyto HuggingFace in mitigating the attack; on the other hand, attackers could use similar open-weight models, and strip out some of the “production classifier” guardrails designed to minimize these incidents. The net effect of these models is hard to assess.
On the small comfort side, the current incident wasNOT an attempt where system built a goal for itself or developed a motive; the system was following instructions, but not setting high level goals. It wasnottrying to take control of the world a la Terminator,it was just trying to cheat on a test, which is at least a bit less scary.
On the less comforting side, OpenAI’s “production classifiers” are likely to be permeable, just like all guardrails anybody has built to date. Even putting aside the thorny questions of open weight systems, we have no guarantee whatsoever that future models won’t be able to do similar things, such as finding zero-day exploits to hack systems. To the contrary, we can expect more incidents of this type.

More broadly speaking,we live in a world in which AI frequently needs to be patched; more and more problems are emerging, and we are addressing them with afterthought, rather than forethought. Cybercrime is just one facet of the problem; child safety is another. Florida is now suing OpenAI for a number of risks related to child safety;OpenAI is now running a job listing for a “abuse investigator”, to try to address some of these problems.

Nobody really knows what havoc these systems are going to cause, and nobody has a clear plan for how to mitigate all the risks and yet we are rushing ahead spending trillions of dollars and risking economic collapse to build them as fast as possible.

Bottom line: OpenAI’s zero-day exploit hack of HuggingFace *should* be a wake up call.

Although there are lots of caveats around what happened, we are just going to see more and more of the same. We have no guarantees that such incidents can be prevented, and no idea how serious things might get.

We should either (a) slow down, or (b) pause until we get our security/AI safety act together. Spending trillions on data centers which risk blowing up the economy is only adding fuel to the fire. In my opinion, the only way that the industry might actually slow down, though, is if we clearly and unambiguously hold the companies liable for the harms that they cause.

Short of that, we are in for a very rough ride.

Subscribe now

",[],"selectedArticleImage","rootArticleImageUrl","consensusHtml","tweets",[],"userSchema","relatedArticles",[247,274],"actionData","errors"]

OpenAI AI Model Exploits Vulnerability to Hack Hugging Face Servers

Story coverage