Anderson Cooper looks pretty worried here, as Jacob Coxon, the ex-OpenAI/ex-Anthropic employee who is the media darling of the moment, tells us that the human species might not be here in five years.

I am here to tell you that Anderson Cooper doesn’t need to worry about that particular scenario, and that you don’t either.

Before I get there, I want to set some things straight. A bunch of people, apparently on the right, are trying to assassinate Coxon’s character. I have seen lies about how long he worked at Anthropic, possibly derived from an inaccurate AI-generated profile of him, and a weaponized thread characterizing him (without real evidence) as a Democratic operative. Someone described him as entry-level employee, which is absurd, given that he had already worked at OpenAI on a technical team for over two years.

In reality, we need a little nuance here; some of what Coxon says is true, some is speculative; some he is in a position to speak to, and some is out of his expertise. (The media too rarely distinguishes).

What I liked about Coxon’s now-infamous thread is that it cast light on what people at OpenAI and Anthropic are thinking. Since he worked there for a total of about three years (mainly at OpenAI, but most recently in Anthropic), he’s presumably in a position to testify to what might be common (presumably not universal) thinking at those companies.

The most important of his thread was the part that went towards state of mind:

rightly accusing the frontier companies of hubris:

Corroborating evidence soon came from Evan Hubinger, a current Anthropic employee:

And others eventually weighed in; OpenAI’s chief scientist actually said something similar. There’s good reason to believe that at least some companies really believe this kind of thing, and also (terrifyingly) that many have come up with justifications for believing that it is ok to continue working on a technology that they themselves appear to be dangerous.

But … there is less reason to believe that people at these companies have broad expertise about how the world works beyond their technical expertise.

Indeed, there is good reason to doubt that so-called Doomers have any clue what they are doing in terms of enacting change in the real world, or even in thinking through the consequences of their own actions. They are fabulous at getting publicity (the Yudkowsky/Soares bestseller, Bostrom’s bestseller, Coxon’s thread, the Pause letter, and so on). And they are fabulous at getting truly massive funding, perhaps close to (if not over) a billion dollars by now.

But in my view, the net effect of the last decade of “Doomer” talk (Coxon appears to be a card-carrying member of that party) has been to make the Frontier AI companies wealthier and more powerful. In so doing they have also aided and abetted a concentration of power that is in itself of highly dangerous. Everytime they scream that AI is going to kill us all, VCs toss in billions more. None of the AI safety work Doomers have sponsored has led to a demonstrated, implemented solution to the core problem. Enabling OpenAI and Anthropic may prove to be one of the worst things humanity has ever done, and they played a big role in that with their constant drama.

Furthermore, Doomers seem to constantly be selling humanity short, as I will discuss below. And (as also discussed below) they also seem to know not the slightest thing about how war works—which matters since they are implicitly or explicitly imagining a war-to-beat-all-wars between humans and machines.

Even if you believed that machines were trying to take over the world (which I don’t, at least not at present nor anytime soon), and that they were superintelligent (also not true yet, though presumably true eventually), and even if you suspended disbelief about unplugging the machines, you still have to think through how it is that the combination of motive and superintelligence would lead to the complete annihilation of the human species.

I have never, ever seen anyone from that world address that latter question with nuance and sophistication. Certainly Coxon has (so far) not done so.

The best attempt I have seen, such as it is, is a book that was wildly popular a year ago, Eliezer Yudkowsky and Nate Soares’If Anyone Builds It, Everybody Dies.

The book has a lot going for it, both as a work of entertaining science fiction, and as a call to arms to get people to take AI risk seriously.

But the part in which it paints how AI might actually extinguish humanity is weak, menadering, and unconvincing. The New York Times Book Review dismissed it as being like scientology.I was kinder in the Time Literary Supplement, and much less dismissive, taking the details of the book far more seriously, but in the end the book’s argument was flaweed, for a multiple reasons.

Here are some excerpts of from what I wrote then, a year ago, all still quite relevant.

The first sums up the Yudkowsky/Soares argument. I presume that Coxon (who hasn’t made his argument explicit as far as I know) is relying on something similar:

The bad news is that Premise 1 (sorry I can’t handle the British spelling) is almost certainly true, though I seriously doubt it will happen by 2030. (Lots of well-known AI researchers like Rich Sutton and Yann LeCun and probably Ilya Sutskever and Fei-Fei Li would agree with me there; we all think that the field needs more breakthroughs before we reach superintelligence.)

That’s already reason for Anderson Cooper to breathe a sigh of relief; we have at least a bit more time before superintelligence arrives than Coxon seemed to allow. (Note that Transformers took 9 years to reach the current state; even if there a new breakthrough now it might take years to fully develop. And we probably need more than one, as I will discuss in a future essay.)

The good news for humanity is that the second and third premises of the Yudkowosky/Soares argument are far, far more shaky. Quoting what I said last year in TLS:

[Note that the Open AI Hugging Face incident was provoked as part of a training exercise, with guard rails partly turned off, and not something that happened organically and spontaneously.]

If the second premise seems unrealistic with respect to machines, the third seems unrealistic with respect to humans:

I sent the review to Yudkowsky, but never heard back. So far I know all my points about human resistance still stand.

Certainly Coxon has not (thus far) added anything substantive exto the argument.

In my view, the chance that AI will eliminate humans in the next five years is all but indistinguishable from zero. Humans are too geographically spread out, too genetically diverse, and too resourceful to simply fall apart altogether. The idea that AI will kill us all in five years is preposterous.

And no, to answer a comment I got on X when I raised these issues, an AI-generated virus is not likely to kill literally all of us, either. When I asked the virologistabout this, she told me “i mean, never say never but the only virus I know that has near total mortality is rabies, and considering it’s not airborne and takes 2-4 weeks to kill you, I don’t think it would take out all of humanity. Ebola doesn’t have 90% mortality like the first outbreak did. It’s closer to 50-60%. Still extremely high but not civilization ending. A pandemic could wipe us out if civilization collapsed...eventually. I’d guess anywhere north of 20-30% mortality would do that. But most pandemic viruses are not that lethal. Even the plague wasn’t that lethal.” In her view, we should be a lot more worried about how AI could cripple civilization than freaking about imaginary AI-created viruses.

Which is not at all to say we are home free.

We really need to make a distinction that I have tried to make many times, betweencatastrophic riskandexistential risk.Existential risk has been defined in the literature as the risk of literal extinction; again I think that’s barely above zero, and that AI doesn’t really change that. (Gamma rays could in principle kill us all, too, but I don’t lose sleep over that.)

Catastrophic risk, on the other hand, is quite real. The possibility of nuclear war remain a huge threat to humanity. (I plan to write about that some point; it is much worse than most people realize.)

And AI does in my opinion significantly elevate the risk of catastrophe (say killing 1%+ of humanity or severely damaging modern civilization ), via multiple paths.

AI elevates the risks of bioweapons attacks by making it easier for terrorists to figure out how to develop and implement such attacks.
AI may conceivably elevate the risks that someone will develop a dangerous new virus or other bioweapon.
AI elevates the risks that disinformation will disrupt democracy, by reducing the cost of generating disinformation and increasing the quality (e.g, with ever-better deepfakes)
AI appears to be elevating the risks of serious cyberattacks that could hobble things like banking or electrical grids.
AI introduces a new vector for authoritarians around the globe; by influencing training sets and so forth, authoritarians can influence LLMs in ways that subtly influence their users.
AI also serves as a back door for surveillance, at a scale beyond what Orwell envisioned.
AI is already harming the educational system by leading students into despair and by seducing them into a faux substitute for learning that is easy to use but that ultimately leaves them with poorly developed critical thinking skills.

We should be focusing on things like these, not fairy tales.

Marcus on AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

",{"_263":208,"_264":292,"_265":293},"a7edc487-e4f2-475c-b4b4-8282294cbd1e",{"_21":292,"_33":294,"_210":295,"_35":296,"_233":297,"_270":297,"_23":240,"_271":298,"_273":299,"_226":42,"_70":-5,"_72":42,"_188":300,"_263":208,"_289":312},"Two Dire Warnings, One from Terence Tao, the Other from Someone Who Just Quit Anthropic","Renowned mathematician Terence Tao has expressed growing concerns about AI's impact on scientific integrity and intellectual progress, particularly regarding intellectual property and the nature of insight. A former Anthropic employee, Jacob Coxon, has also voiced anxieties about the safety and ethical considerations within leading AI labs, warning of potential catastrophic harms. The article highlights a significant shift in perspectives, moving from initial optimism to deep-seated worries about the future of science and AI development.","Lots to Consider","https://substackcdn.com/image/fetch/$s_!Uw_q!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9080a929-323b-4e72-b47f-7b5b942d9fd0_963x1276.jpeg","2026-09-09T02:18:51","https://garymarcus.substack.com/p/two-dire-warnings-one-from-terence",{"_21":301,"_191":302,"_193":119,"_70":303,"_219":28,"_220":310,"_222":311},"8b40407a6d70178f8f93a815477be4f4487b86df33da10dcd354c6106f345b24","990141b96cdd06229ad5a1b2940fef9f731d0a5ed244b6967d8e6407612b3a71ef398bcb5f9908df584f8e6258bae61d1adc496430252cad95be82058b3ea520",[304,305,306,308,309],[226,294],[37,297],[282,307,42,284],"30023:983dd797fe72ad62fc8e3ece7eed1f126550972885c8f835d9af5528dad655a3:a7edc487-e4f2-475c-b4b4-8282294cbd1e",[216,42],[218,28,42],"Two Dire Warnings, One from Terence Tao, the Other from Someone Who Just Quit Anthropic\nLots to Consider\n\n\nnostr:naddr1qvzqqqr4gupzpxpa67tluu4dvt7gu0kw0mk37yn92ztj3pwglq6ant649rddv4drqqjxzdm9v33ngwph94jnge3j956rwdtr943rgc3595urywpjxgungcmzvsck20tjy2d ",1788920331,"

Part I: Terence Tao

Back in fall of 2024, Terence Tao, perhaps the most respected living mathematician, was eager to learn about what AI could and could not do. Although he was skeptical of the then recently-released o1, he was optimistic about a future in which humans and machines co-existed. Matteo Wong had great interview with Tao about this inThe Atlantic:

Wong spoke to Tao by phone and summed up Tao’s openness to new futures this way:

Marcus on AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Lately, as I notedin a Substack here in August, Tao has been raising questions. At the end of July he gave a great lecture, with this among his slides.

He’s clearly been thinking about this ever since. By now, it is clear that his views have radically shifted.

Three posts of his from the last two days illustrate:

The first (yesterday) notes that solving puzzles is not the same as coming up with new insights:

The second (also yesterday) is in some ways an argument about good intellectual taste, and expresses deep concerns about the consequences of AI for math.

It is also a plea for transparency:

Thefinalone (all appeared on mathstodon) came out today and appears to allude to the OpenAI-NYU-Anthropic Navier-Stokes controversy, in which OpenAI rushed to scoop Alpöge and Buckmaster,spending $22.5 million in the process, possibly using their data. (In vague, evasive words OpenAI wrote that “we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”)

Tao starts by again discussing the question of good intellectual taste, as background. As a scientist who has worked in other areas, I completely resonate with his opening framing.

That last sentence is a truly dire warning, about a potentially tragic world.

And indeed, as Tao implies, there will be fallout in other fields as well.

Fat chance of AI “curing” cancer if nobody trusts the AI companies not to steal their IP. These words from OpenAI’s Chief Research Officer are hardly reassuring:

As I noted

I have no idea what the solution is here. But I desperately hope that Terence Tao’s warnings about how all of this might impact science will be heeded.

Part II: Jacob Coxon, who just resigned from Anthropic

Just I was finishing up,thisjust came in, from Jacob Coxon, who just left Anthropic.

I don’t find it quite as compelling, and it is focused on future harms with too little discussion of current harms, but it is spreading like wildfire and worth reading.

I happen to disagree with him around timing, and I would argue that Coxon is exaggerating what AI is likely to do anytime soon, But it is nonetheless overall a disconcerting (and plausible) firsthand perspective on the transparently self-indulgent and dangerous thought processes in two of the leading frontier labs:

Anthropic’s Alignment Science lead wrote this

I can’t say I find this comforting.

AlthoughI don’t think extinction is likely, catastrophic harm is certainly possible, and it is indeed clear that nobody has a serious plan. The distinction of “getting there first” is probably fairly irrelevant, if others will soon follow.

Perhaps the only thing that could really help here on the technical side would be a different foundation than LLMs (which continue to seem utterly incorrigible) and neither company seems to be taking that notion seriously.

The end of open science? Or worse? Neither scenario is pretty.

Please consider becoming a subscriber, free or paid, and join over 115,000 others.

",[],"selectedArticleImage","rootArticleImageUrl","consensusHtml","tweets",[],"userSchema","perspectivesColumns","intro","relatedArticles",[266,293],"actionData","errors"]

Former Anthropic Employee Jacob Coxon Warns of Catastrophic AI Risks

Cobertura da história