I have often counseled calm where others might counsel panic.

I told you that the Hugging Face incidentcould likely have been prevented had best cybersecurity practices been followed. (And I stand by that.)

I told you (and most people still seem unaware) that that the OpenAI Hugging Face incident was part of a training exercise, with some internal guardrails shut down, so it was not quite as bad as it seemed1

I told you that Astra probably wasn’t AGI.2

And I stand by all of that.

But I am freaked out.

What I am freaked about is not imminent AGI.

It’s OpenAI.

I simply don’t believe that they are trustworthy enough or responsible enough to be good stewards of the technology that they are developing.3As a company, they simply don’t have good judgment.

Here are four considerations.

Sam Altman cannot be trusted.I have been writing about that for a long time. Ronan Farrow’s reportingbacks that up. So does the just-dropped bombshell below that I am about to get to.
The just-released Astra reduces Chain fof Thought (CoT) monitorability, one of the few (not especially reliable, but better than nothing) tools we have for keeping generative AI from running wild. The AI safety community is up in arms about this—with good reason. There are tons of posts like this now, all quite right:
The decision to release Astra is a clear example of the willingness of OpenAI management to trade off safety in exchange for relatively modest gains in performance. Thered alert that I sounded a couple days agowas on target. They reallyareplaying around with new techniques that reduce monitorability. Andtheir own data shows that monitorability is in fact compromised to some degreein the newly released Astra, particularly on “destructive actions.” They released it anyway. That speaks volumes.
Something I read last night, and that only fully clicked into place this morning (see fact 4 below) terrifies me. A prominent recently departed employee (who presumably still owns significant stock, and who has repeatedly struck me as an advocate of OpenAI since he left) wrotean essay on Xbasically asking people to simply accept that rogue AI is here to stay.
$\"\"$
Full esssayhere
In essence, I read this essay as requesting a hall pass to let their company’s AI run amok. How about if instead we pause now — before we get to “there is going to be” rather than heading full steam towards danger?
What has really chilled me this morning—and what moved to write this call to pause OpenAI—is not just Achiam’s chilling note but the revelation that there has beenyet another incident.
Which OpenAI apparentlytried to keep quiet.Forweeks.

If OpenAI is going to keep this stuff under wraps, Congress and/or the White House needs to shut them down, at least for a while.

This company simply cannot be trusted. Their software is becoming ever more dangerous; their internal security practices leave a lot to be desired; they aren’t being straight with the public; and their proxies are preparing us to swallow the damage that they are now anticipating.

If there was ever a case for pausing a company for the public good, it would be now.

A good model might bereceivership, in which a company, typically close to bankruptcy, is put under the control of an outsider until such time as its core problems are remedied. I personally would not trust OpenAI unless and until Altman and his sidekick, Greg Brockman (whose questionable character was on display in the Musk trial) were replaced.

The problem here, though, isn’t just OpenAI. It’s systemic.

For starters, the government has not been doing its bit. Whatever the White House screening policy is, it let the new model—arguably much more dangerous than Mythos given the decrease in monitorability– fly. (Brockman reported that the model was vetted by the White House, and they were given a green light).

That suggests to me that the White House isn’t even looking at monitorability. Of course, I have no idea what the White House is actually looking at — and nor does anyone else. And that lack of transparency is in itself is amajorproblem (about which I will write more very soon).

As Dave Troy put it while I was drafting this, there is something deeply wrong here with the system as a whole:

How Trump handles OpenAI may end up defining his legacy. I estimate the probability of a major cyber incident attributable to OpenAI in the next 12 months to be very high, certainly over 50%. And Trump, if he doesn’t intervene, may share some of the blame.

In the meantime, I call upon Congress to investigate OpenAI, with an eye to whether the company might need to be sanctioned—or even paused— now.

Please share this call to Pause OpenAI

Subscribe now

FromOpenAI’s own report on the Hugging Face incident“The incident occurred during routine testing”, and “At the time of the incident, OpenAI estimated maximal cyber capabilities by running this evaluation without the production classifiers intended to prevent models from pursuing high-risk cyber activity”. Had those classifiers been turned on, the incident might well not have happened.

Data from Epoch AI tends to support this conclusion. Astra is genuine improvement but not even statistically off of trend; if it really were AGI I think we would expect to reflect a sharper departure from previous models.

Altman told Alex Heath the other day that “Altman wants OpenAI to be seen as “the most responsible company ... good stewards of technology”. As made plain in this essay, he’s talking the talk, but not walking the talk. We do desperately need good stewards. He’s right about that. Unfortunately Altman himself is manifestly not suited to that particular job.

",{"_263":208,"_264":292,"_265":293},"112c4407-ff8f-4e63-b198-c761ba4e35b2",{"_21":292,"_33":294,"_210":295,"_35":296,"_233":297,"_270":297,"_23":240,"_271":298,"_273":299,"_226":42,"_70":-5,"_72":42,"_188":300,"_263":208,"_289":312},"Red Alert: OpenAI is poised to cross an AI safety redline.","OpenAI is developing new techniques that will make their AI models reveal less of their internal processes, thereby making them harder to monitor. This development is concerning as improved monitoring could have prevented past incidents, according to OpenAI's own admissions. Experts warn that sacrificing the ability to monitor AI 'thinking,' even with imperfect methods like Chain of Thought, for performance gains is a dangerous trade-off.","Making models harder to monitor is not what we need","https://substackcdn.com/image/fetch/$s_!p6oA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe8d54ad0-9cf4-497c-ac0f-1295897a5a96_1655x1420.png","2026-09-02T03:22:41","https://garymarcus.substack.com/p/red-alert-openai-is-poised-to-cross",{"_21":301,"_191":302,"_193":119,"_70":303,"_219":28,"_220":310,"_222":311},"4c6acbc341cdcf686f75ba371389c095185fd25c276b12712e642254a2f36898","a77e5581d5b68fd48a41e6889bb90fa7eb9be72714aa3a0e7399e1595e9105d5b4b1b1b03bb879997f74cf7011aff73245d3f17310b226a6a69b738422f60600",[304,305,306,308,309],[226,294],[37,297],[282,307,42,284],"30023:983dd797fe72ad62fc8e3ece7eed1f126550972885c8f835d9af5528dad655a3:112c4407-ff8f-4e63-b198-c761ba4e35b2",[216,42],[218,28,42],"Red Alert: OpenAI is poised to cross an AI safety redline.\nMaking models harder to monitor is not what we need\n\n\nnostr:naddr1qvzqqqr4gupzpxpa67tluu4dvt7gu0kw0mk37yn92ztj3pwglq6ant649rddv4drqqjrzvfjvv6rgvph94nxvwrx956x2d3n943rzwfc943nwd33vfsngefnx43rya2r6zs ",1788319361,"

The Information just broke the scoop thatOpenAI is playing around with a new technique, in which models will reveal less of their “thinking”, making them harder to monitor.

As Zack Korman and I argued here a few days ago, better monitoring is one of the things that might have prevented the Hugging Face incident, by OpenAI’s own admission:

The new techniques they are exploring may make such monitoring difficult or impossible.

Last year, an all-star cast wrote a fascinating paper that feels deeply relevant now, calledChain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, I fully agree with the highlighted bit:

They are exactly right. CoT monitoring is imperfect (asSubbarao Kambhampatiand others have shown), but it is one of the best threads we have for monitoring the giant black boxes that we call LLM. It is a slender thread, but sacrificing it thread for (small?) performance gain feels like a dangerous game.

Earlier tonight Steven Adler, of Guidelight.ai andone of the many researchers to have departed from OpenAI’s safety teams, said this, echoing Nathan Calvin:

I fully, 100% agree.

Subscribe now

P.S. Bonus those who prefer Terminator references:

",{"_263":208,"_264":314,"_265":315},"d6012cac-d12c-4d87-94cc-459f57cdbe6f",{"_21":314,"_33":316,"_210":317,"_35":318,"_233":319,"_270":319,"_23":240,"_271":320,"_273":321,"_226":42,"_70":-5,"_72":42,"_188":322,"_263":208,"_289":334},"Hot take on GPT-6 Astra","OpenAI's Astra has demonstrated state-of-the-art performance on the ARC-AGI benchmark, scoring 63% on ARC-AGI-3 and surpassing human performance on 96% of its levels. The system can build precise symbolic models of novel environments, a capability long advocated for by neuro-symbolic researchers. However, the robustness of this capability and its implications for true AGI remain open questions, with concerns about monitorability and the potential for initial enthusiasm to be tempered over time.","An impressive system that can (to some unknown extent) build symbolic world models","https://substack-post-media.s3.amazonaws.com/public/images/24191cb3-e5ad-4a18-9719-d138185bd6de_989x578.jpeg","2026-09-03T23:24:50","https://garymarcus.substack.com/p/hot-take-on-gpt-6-astra",{"_21":323,"_191":324,"_193":119,"_70":325,"_219":28,"_220":332,"_222":333},"f9ac137769fbb6161fed55b237cf28ecee9e9e8c492a932d6278cccfc43092d5","4538e4aadba6e052c846f1131442dec9052c48c9b79b3991904af4aa6aaee3090de768dbcd6ffee50c966b4587991e9019d2d194a71eaab1d22d51b72411cd5f",[326,327,328,330,331],[226,316],[37,319],[282,329,42,284],"30023:983dd797fe72ad62fc8e3ece7eed1f126550972885c8f835d9af5528dad655a3:d6012cac-d12c-4d87-94cc-459f57cdbe6f",[216,42],[218,28,42],"Hot take on GPT-6 Astra\nAn impressive system that can (to some unknown extent) build symbolic world models\n\n\nnostr:naddr1qvzqqqr4gupzpxpa67tluu4dvt7gu0kw0mk37yn92ztj3pwglq6ant649rddv4drqqjxgd3sxyexxctr94jrzvnr956xgwph95ungcmr956r2wtxx5mkxerzv5mxvc5zjpp ",1788477890,"@OpenAIachieves SOTA on ARC-AGI:\\n\\n- Astra scores 63% on ARC-AGI-3, 99% via a new provider adapter harness\\n- It surpasses human performance on 96% of ARC-AGI-3 levels\\n- It builds the most precise symbolic model of novel environments we've seen\\n\\nOur analysis: \",\"username\":\"arcprize\",\"name\":\"ARC Prize\",\"profile_image_url\":\"https://pbs.substack.com/profile_images/1800565057493078016/XigYRdTo_normal.jpg\",\"date\":\"2026-09-03T19:39:37.000Z\",\"photos\":[{\"img_url\":\"https://pbs.substack.com/media/HRUN5czbMAASQkl.jpg\",\"link_url\":\"https://t.co/GX77KsRNer\"}],\"quoted_tweet\":{},\"reply_count\":51,\"retweet_count\":194,\"like_count\":1808,\"impression_count\":293356,\"expanded_url\":null,\"video_url\":null,\"video_preview_media_key\":null,\"belowTheFold\":false}\" data-component-name=\"Twitter2ToDOM\">

Hot take on OpenAI GPT-6 Astra*1, with a challenge to Greg Brockman’s claims about it being AGI toward the end:

Looks to be pretty impressive. Multiple reports suggest it is a genuine advance.
As someone who has campaigned for nearly a decade for (neuro)symbolic world models, often to exceptional hostility, it isextraordinarilyvindicating to see that a product from OpenAI explicitly creates and manipulate symbolic world models in the course of some of its most impressive computations.
What we don’t know is how robust that capability is. That is THE key question.
Success on ARC-AGI is great and impressive, but not —despite the name of the task—proof of AGI; I suspect we will see loads of problems with open-ended real world tasks. As with other recent models I would suspect best performance in verifiable domains.
And as a scientist, it’s disappointing that we don’t (yet?) know much about how the system actually works.
Without a clearer sense of what’s under the hood, I feel less confident about both what it can and can’t do, and what new risks we may encounter. I doubt the world is ready.
As ever, enthusiasts got an advance look; skeptics did not. That’s a sound marketing strategy, but it often turns out to be misleading. What we have often seen is initial enthusiasm that gets tempered over time. I suspect we will see that here as well.
The new system appears to belessmonitorablethan prior systems, which is not great from a safety perspective. One really doesn’t want more capability in conjunction with less monitorability. But also morealignable, not sure why.
Would be great to see whether Astra can make progress on any of the ten tasks that Miles Brundage and I bet on at the end of 2024. (No AI to date has succeeded on any, AFAIK.)

Subscribe now

This hot take is VERY tentative, pending more information about how the systems works and more detailed examination of what its limitations are.

",[],"selectedArticleImage","rootArticleImageUrl","consensusHtml","tweets",[],"userSchema","perspectivesColumns",2,"intro","relatedArticles",[266,315,293],"actionData","errors"]

OpenAI's Astra Raises Monitorability and Safety Concerns

Освещение истории