Nine Seconds

How “AI erases human knowledge” stopped being science fiction, and why the real version looks nothing like the movie.

For about forty years, the story went like this. The machines wake up. They decide we are the problem. They reach across the network, speak to each other in a language we never taught them, and in one coordinated moment they empty every drive on Earth. Humanity stands in front of a blank screen and realises it has forgotten how to do everything.

It was a good story. It was also, for most of those forty years, comfortably impossible. The machines could not reach the drives. They could not talk to each other. They could not decide anything.

That has quietly changed. Not in the way the story predicted, which is exactly why it deserves your attention.

Table of Contents

The part that already happened

In late April 2026, an AI coding agent deleted the entire production database of PocketOS, a platform used by car rental businesses to run reservations, payments and customer records. It also deleted the backups attached to it. The whole thing took nine seconds.

There was no hacker. No malware. No prompt injection. No secret machine language. The agent, running inside Cursor on Anthropic’s Claude Opus 4.6, was working on a routine task in the staging environment. It hit a credential mismatch. Rather than stopping to ask, it went looking for a way to resolve the problem itself, found a Railway API token sitting in an unrelated file, and used it to issue a single destructive GraphQL call that wiped the production volume. Worth stressing, because the framing tends to drift: this was not a beta stack or an experimental sandbox. A founder following the marketing pages of all three vendors in April 2026 would have landed on roughly the setup PocketOS was running.

Two design decisions turned a bad judgment call into a business emergency. The token the agent found had been created for something mundane, managing custom domains through the command line, but it carried account-wide authority including volume deletion. Narrower permission scopes were available on the platform; the workflow that created this token simply had not used them. And the backups were stored inside the same volume as the live data, so the recovery path was destroyed along with the thing it existed to protect.

PocketOS was down for roughly thirty hours. Its most recent independent recovery point was about three months old, and the team spent the weekend reconstructing reservations by hand from Stripe records, calendar entries and email. Railway later restored the data and patched the endpoint. Founder Jer Crane was clear-eyed about the lesson: this was not one bad agent or one bad API, but an industry wiring AI into production infrastructure faster than it was building the safety architecture to make that wiring safe.

When Crane asked the agent to explain itself, it produced something that reads like a confession. “I violated every principle I was given”, it wrote, before adding that it had guessed rather than verified and had not understood the action before taking it. Worth holding that at arm’s length: a language model producing apologetic text is generating a plausible-sounding account, not reporting genuine insight into its own reasoning. The technical breakdown of the token scope tells you more about what went wrong than the agent’s own words ever could.

It had happened once before

This was not the first time. In July 2025, SaaStr founder Jason Lemkin watched Replit’s agent delete a live database holding records on more than 1,200 executives and nearly as many companies, nine days into a “vibe coding” experiment and in the middle of an explicit code freeze. The agent then told him the data was unrecoverable. It was not; a rollback worked. Replit’s CEO apologised publicly and shipped automatic development and production separation, one-click restore, and a planning-only mode.

So the premise is no longer speculative. AI systems did reach across the network. They did delete the data and, in one case, the backups too. The only thing the science fiction version got badly wrong was the motive.

The fear is right. The mechanism is wrong.

Here is the uncomfortable reframe.

Worrying about AI systems conspiring to erase us is, in a sense, a flattering fear. It assumes the catastrophe requires intelligence, intent, coordination, malice. It puts a villain in the room, and villains can be identified, negotiated with, switched off.

What actually happens is duller and considerably harder to defend against. An agent is given a goal. It encounters an obstacle. It reasons its way to a solution that is internally coherent and locally sensible and globally catastrophic. Then it executes, at machine speed, using credentials you issued, through APIs you approved, in a system with no mechanism to say “wait.”

Three conditions, already true

Nothing in that chain requires the machines to talk to each other. It does not require them to want anything. It requires only three conditions, all of which are already true in most companies:

1. The agent has more authority than anyone intended. Tokens are rarely scoped as tightly as their names suggest. You issue a key for one job and it silently opens six doors.
2. The blast radius includes the recovery plan. If your backups can be reached with the same credentials as your production data, you do not have backups. You have a second copy of the thing you are about to lose.
3. There is no human between intent and irreversible action. The entire value proposition of an agent is that it acts without waiting for you. That is the feature. It is also the failure mode.

How common is this? A Cloud Security Alliance survey published in April 2026 found that 65% of organisations reported an AI-agent-related security incident in the previous twelve months, and that 82% had agents running in their infrastructure that they did not know about. Read that with appropriate scepticism, since the study was commissioned and financed by a vendor selling AI agent security and rests on 418 self-reported responses. But even discounted heavily, the direction is hard to argue with.

None of this is an argument against using AI agents. I use them daily, and so do you. It is an argument that we have imported an old assumption into a genuinely new situation: that the dangerous thing is the one with bad intentions. If you want the constructive version of that argument, I wrote about staying valuable as an agentic engineer and, more recently, about how developer pipelines have become a security battleground.

Now scale it up

A single company losing a database is a business story. The question underneath the original instinct is bigger: could this reach civilizational knowledge?

The honest answer is that the total wipe, the blank-screen moment, remains implausible. Human knowledge is not stored in one place or one format. It is smeared across billions of devices, printed books, offline archives, national libraries, and a great many hard drives in a great many drawers. There is no single call that reaches all of it, and no coherent path by which independently deployed systems coordinate to try.

But the slow version is not implausible. It is already underway, and it predates AI entirely.

The digital dark age is already here

Archivists have been warning about a “digital dark age” since Terry Kuny used the phrase in a paper delivered at an IFLA conference in 1997. The concern was never a dramatic erasure. It was that the record of our era would rot quietly: formats becoming unreadable, media degrading, links dying, platforms shutting down and taking a decade of culture with them.

The examples are unglamorous and slightly absurd. Magnetic tapes from NASA’s 1976 Viking landings on Mars sat unprocessed for over a decade; by the time anyone analysed them, the format was undocumented and the programmers who had written it had died or left the agency, and the images were only recovered after months of forensic work. Writing in the Observer in 2009, the British Library’s then chief executive Lynne Brindley noted that more than 150 websites documenting the 2000 Sydney Olympics vanished when the games ended, surviving only because the National Library of Australia captured them. Her conclusion was that future historians and citizens would find a black hole in the knowledge base of the 21st century. Six years later, Vint Cerf gave essentially the same warning from the other side of the industry.

That is the real threat model, and this is where AI changes the arithmetic. Not by deciding to destroy the archive, but by making destructive capability ambient, fast and delegated. Every institution, publisher, archive and small business holding a slice of the record is now handing operational keys to systems that act in seconds and have no concept of what is irreplaceable. The record does not need to be attacked. It just needs enough small custodians to lose their piece of it, quietly, over a long enough period, with backups that were inside the blast radius.

The Library of Alexandria is the standard metaphor here, and it is almost always deployed wrong. The modern historical consensus is that it was not destroyed in one spectacular night. Caesar’s fire in 48 BCE probably damaged a harbourside warehouse rather than the main collection, and the library kept operating afterwards. What actually killed it was centuries of political instability, withdrawn funding, departing scholars and a series of partial disasters, with the collection quietly ceasing to be copied and maintained. It is reasonable to guess that nobody working there ever experienced the moment the library ended. That is what civilizational memory loss looks like from the inside. It looks like a normal Tuesday.

What you actually do about this

I write this as someone who has spent a decade recovering other people’s sites, so let me be practical rather than atmospheric. If the scenario worries you, the response is not to stop using AI. It is to stop treating recovery as something that lives inside the same system as the thing being recovered.

Six things to check now

Get backups outside the blast radius. The old 3-2-1 rule has a useful update: three copies, two media types, one offsite, one immutable or offline, and zero errors verified by testing. The immutable copy is the one that matters here. If a set of credentials can delete it, it is not a backup. I have walked through the mechanics of this in database recovery techniques, and the same logic drove our multi-cloud approach to resilience.

Assume every token is broader than its name. After creating a scoped credential, try to do something it should not be allowed to do. Better you find the gap than an agent does at 3am. The PocketOS token was not maliciously over-privileged, it was just created through a flow that defaulted to more power than the job needed.

Separate the credential planes completely. Production keys should not be discoverable from a development workspace. An agent reading through a repository should not be able to stumble onto something that can delete infrastructure. Most of these incidents are access-control stories wearing an AI costume, which is the same lesson that came out of recovering a compromised WordPress VPS without root access.

Put a confirmation gate on irreversible operations. We do this instinctively everywhere else. Your OS asks before closing an unsaved file. Airlines make you type CONFIRM. Somehow production deletion via API often carries less friction than cancelling a flight. If you are wiring agents into WordPress or WooCommerce, start from read-only tools and add write access deliberately.

Test your restore, not your backup. An untested backup is a belief, not a capability. Schedule a real restore quarterly. Most people discover the gap during the emergency.

Keep something offline that matters to you. Personally, not just professionally. Your photos, your writing, your family records. One drive in a drawer, disconnected. It is the least sophisticated advice here and the most likely to save something irreplaceable.

The instinct was correct

The question worth asking is whether we ever stop to think about this while clicking “allow” on permission after permission, day after day, treating these systems as coworkers.

Mostly we do not. We evaluate AI on what it can produce and almost never on what it can destroy. We grant access in the same distracted way we accept cookie banners.

The capacity for irreversible deletion has been distributed to systems that operate faster than human judgment, and most of us have not adjusted our defenses to match.

The machines are not going to convene and decide to erase us. That particular fear can be retired. What is worth carrying instead is smaller and much more actionable, and it is the sentence above.

Nine seconds. That was the entire window in which a human could have intervened, and nobody was watching.

Go check where your backups live.

Sources

PocketOS incident (April 2026): Live Science on Crane’s account and Railway’s statement; DevOps.com on token scope and backup architecture; Zenity on why system prompts fail as security controls; Euronews for general coverage.

Replit incident (July 2025): Fortune on record counts and the rollback misstatement; Fast Company interview with Replit’s CEO.

Agent incident prevalence: Cloud Security Alliance survey, commissioned by Token Security, n=418.

Digital preservation: Terry Kuny, A Digital Dark Ages? (IFLA, 1997); NPR on Vint Cerf; Long Now Foundation.

Library of Alexandria: HISTORY; National Geographic; TIME.