We must recall open-ended AI agents with internet access from the market, now
Every minute we wait risks disaster
TL;DR
- Incidents at OpenAI and Anthropic indicate current AI agents are untrustworthy.
- The AI industry is not being safe enough, operating like a startup with insufficient safety measures for dangerous systems.
- Potential risks from AI, such as loss of control, could be catastrophic, exceeding events like nuclear power plant meltdowns.
- There is a lack of robust internal controls and redundancies for AI safety compared to other high-risk industries.
- The administration's actions on AI disclosure are insufficient, and a temporary recall of dangerous AI technology is overdue.
Breaking new fromThe New York Times, the latest ofmany agent-caused incidents that are happening with frightening regularity, but this time from Anthropic and fairly serious:
I have long felt that OpenAI is handling this inadequately. Seeing the same kind of incidents at Anthropic makes it absolutely clear that the current generation of synthetic agents simply cannot be trusted. Until they can be fixed, they should be removed from the market, just like a car with defective brakes.
§
Ezra Klein’s new interview with recently departed AI employee David Robinson only furthers my sense that these companies are in wildly over their heads:
Quoting in part:
And I don’t think that we or our peers — really, anyone in the industry — are being safe enough. I think OpenAI and its peers are now producing a technology that is more capable and poses more risk than what was being made even six months ago.
I’m not a scientist. I’m a writer. What I know is what the execution environment looks like for our safety work, and we’re operating — and I believe the industry is operating — like a start-up still, more so than makes sense. Not maybe completely like a brand-new start-up, but we’re too close to that end of the spectrum for really dangerous systems that could pose risks — loss of control is one example. If that did happen, we’re talking about a harm that’s much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility.
Now, some of this is known, right? OpenAI has publicly reported on safety problems. Obviously, Hugging Face, but also other ones, including more recently. And Anthropic, by the way, also has reported, including an instance in which their safeguards were accidentally misconfigured.
So I think people do have some evidence already externally that things are not as they ought to be. But I also think if you were watching from the outside, you might imagine that we have a more robust safety setup than we actually do.
§
The Trump administration’s anemic request for more disclosure is not enough, akin to blandly asking criminals to file monthly reports regarding which crimes they have committed.
Each day the administration fails to take stronger action is a mistake, inviting worse problems. It’s well past the point at which they should be imposing a temporary recall on an obviously dangerous technology.
When something truly bad happens, the White House, and not just the tech companies, will own it.