Breaking news: U.S. banks are taking hits from AI agents
Who knows what piece of infrastructure might be attacked next.
TL;DR
- U.S. banking executives are alarmed by AI-driven attacks on their systems.
- Attacks on refrigeration systems in France and a wave of cyberattacks in Japan have been reported.
- Hackers are using existing AI models, not just state-of-the-art ones, for malicious purposes.
- Generative AI is a powerful tool for cyberattackers, enabling phishing and deepfake creation at low cost.
- Large language models are vulnerable to jailbreaking and prompt-injection attacks.
- AI-generated code can introduce new security vulnerabilities.
- A new U.S. administration's deregulation efforts may reduce enforcement and investigations.
— Sam Altman, October 3
Breaking news from Madison Mills at Axios:
Full articlehere. “Findings have alarmed banking executives, who are racing to restrict access to their systems as AI moves at a velocity faster than financial institutions can respond”. Day after scenarios!
§
Meanwhile, that’s hardly the only news of this sort. On October 11, there wereattacks on refrigeration systems in Francethat may have been facilitated by agents. There is also a growing, ongoingwave of cyberattacks in Japan. To say nothing of the months of attacks from OpenAI and others on places like Hugging Face and the Australian government.
YesterdayThe Informationreported ominously on what a Latin American hacking group called Cybercartel is up to:
This is disconcerting not just because hackers are stepping up their malicious use of agents, but also because they don’t need state-of-the-art “frontier” models to cause mayhem.
This should serve as a wakeup call for those who focus all their attention on regulating “frontier models”, often measured in terms of how much compute they use. Even if we were to stop the development of future frontier models, we would still face serious problems from the misuse of existing models.
Clearly, regulatory efforts should not be focused solely on the frontier, nor solely focused on concerns about what future models might do.Currentmodels arealreadycausing serious problems across society, in domains ranging from education to politics (via deepfakes and wholesale disinformation) to banking. Ignoring all that entirely in favor of speculative futures would be a mistake.
§
At the beginning of 2025I warned in Politicothat we were about to face a perfect storm in cybersecurity, for four reasons.
First, generative AI, rising in popularity and declining in price, is a perfect tool for cyberattackers. Although it is unreliable and prone to hallucinations, it is terrific at making plausible sounding text (e.g., phishing attacks to trick people into revealing credentials) and deepfaked videos at virtually zero cost, allowing attackers to broaden their attacks. Already, a cybercrew bilked aHong Kong bank out of $25 million.
Second, large language models are notoriously susceptible to jailbreaking and things like “prompt-injection attacks,” for which no known solution exists.
Third, generative AI tools are increasingly being used to create code; in some cases those coders don’t fully understand the code written, and the autogenerated code has already been shown in some cases to introduce new security holes.
Finally, in the midst of all this, the new U.S. administration seems determined to deregulate as much as possible, slashing costs and even publicly shaming employees. Federal employees who do their jobs may be frightened, and many will be tempted to look elsewhere; enforcement and investigations will almost certainly decline in both quality and quantity, leaving the world quite vulnerable to ever more audacious attacks.
Well, here we are.
§
Here are three further predictions:
No matter how much chaos generative AI causes, the AI industry will not take real responsibility
The Trump administration will do little to hold them responsible.
Things will get worse.