Two Dire Warnings, One from Terence Tao, the Other from Someone Who Just Quit Anthropic

Lots to Consider

Part I: Terence Tao

Back in fall of 2024, Terence Tao, perhaps the most respected living mathematician, was eager to learn about what AI could and could not do. Although he was skeptical of the then recently-released o1, he was optimistic about a future in which humans and machines co-existed. Matteo Wong had great interview with Tao about this inThe Atlantic:

Wong spoke to Tao by phone and summed up Tao’s openness to new futures this way:

Marcus on AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Lately, as I notedin a Substack here in August, Tao has been raising questions. At the end of July he gave a great lecture, with this among his slides.

He’s clearly been thinking about this ever since. By now, it is clear that his views have radically shifted.

Three posts of his from the last two days illustrate:

The first (yesterday) notes that solving puzzles is not the same as coming up with new insights:

The second (also yesterday) is in some ways an argument about good intellectual taste, and expresses deep concerns about the consequences of AI for math.

It is also a plea for transparency:

Thefinalone (all appeared on mathstodon) came out today and appears to allude to the OpenAI-NYU-Anthropic Navier-Stokes controversy, in which OpenAI rushed to scoop Alpöge and Buckmaster,spending $22.5 million in the process, possibly using their data. (In vague, evasive words OpenAI wrote that “we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”)

Tao starts by again discussing the question of good intellectual taste, as background. As a scientist who has worked in other areas, I completely resonate with his opening framing.

That last sentence is a truly dire warning, about a potentially tragic world.

§

And indeed, as Tao implies, there will be fallout in other fields as well.

Fat chance of AI “curing” cancer if nobody trusts the AI companies not to steal their IP. These words from OpenAI’s Chief Research Officer are hardly reassuring:

As I noted

§

I have no idea what the solution is here. But I desperately hope that Terence Tao’s warnings about how all of this might impact science will be heeded.

Part II: Jacob Coxon, who just resigned from Anthropic

Just I was finishing up,thisjust came in, from Jacob Coxon, who just left Anthropic.

I don’t find it quite as compelling, and it is focused on future harms with too little discussion of current harms, but it is spreading like wildfire and worth reading.

I happen to disagree with him around timing, and I would argue that Coxon is exaggerating what AI is likely to do anytime soon, But it is nonetheless overall a disconcerting (and plausible) firsthand perspective on the transparently self-indulgent and dangerous thought processes in two of the leading frontier labs:

Anthropic’s Alignment Science lead wrote this

I can’t say I find this comforting.

AlthoughI don’t think extinction is likely, catastrophic harm is certainly possible, and it is indeed clear that nobody has a serious plan. The distinction of “getting there first” is probably fairly irrelevant, if others will soon follow.

Perhaps the only thing that could really help here on the technical side would be a different foundation than LLMs (which continue to seem utterly incorrigible) and neither company seems to be taking that notion seriously.

§

The end of open science? Or worse? Neither scenario is pretty.

Share

Please consider becoming a subscriber, free or paid, and join over 115,000 others.