The Borg Internet Hypothesis

During a phone call to NVIDIA CEO Jensen Huang while Huang was at the All-In Podcast’s All-In Summit Monday, President Trump said, “the robots will not be taking over the world.” Of the discourse in the last few days, this is one the most indisputable things that has been said. Robotics, as a discipline, is not yet a major concern. The statement, while likely said in jest, misses the actual risk entirely. So what, specifically, are folks in the AI industry freaking out about? What would a catastrophic loss of control event actually look like?

To talk about this, let's go all the way back to CompSci 101. I had a professor who did the Peanut Butter and Jelly Sandwich Exercise: they gathered the requisite ingredients for a PB&J, and then asked a student to give them instructions on how to make a sandwich. Students started with “Put peanut butter on the bread” so the professor put the sealed jar of peanut butter on top of the sealed bag of bread. Hilarity ensued from there. My favorite was the plastic bread bag covered in peanut butter. The instructions were followed perfectly, but it showed students the value of defining variables and context in your programs. The computer, without being explicitly told, does not know what peanut butter, jelly, bread, a sandwich, a knife, etc. are or how they interact without being explicitly told.

Large Language Models (LLMs) partially unlock context. An LLM can determine the probability of what action you mean in the context of its training data. If it's seen peanut butter put on bread the same way a majority of times, it has a higher probability of getting it right. The challenge is what else is included in that context.

Let's take a problem that can be more morally problematic than PB&J. I'm a big fan of Korean R&B music, high-quality digital audio, and of owning my files (more on that later). This is a weird niche. The overlap in the Venn diagram of places that sell lossless music files and Korean music to the US market is basically Qobuz and no other examples. Without knowing this, I might prompt an LLM to tell me how to get lossless audio files for a new Korean album. Without guardrails, most modern AI Agents will find the most probabilistic answer on the internet. If you've not been on the internet in thirty years, let me bring you up to speed: we got the internet and a non-trivial amount of people decided it was for stealing music. The practice of adding guardrails to LLMs is, at the very least, ensuring these models are not telling you to do things that are blatantly illegal, no matter how common they are online. LLMs today, luckily, will tell you about Qobuz, before they tell you about pirating, but a few will still give you enough information to dip your toe into piracy, if you're into that kinda thing.

This is the big challenge with training data and AI's context. It doesn't matter if you train AI on the greatest published works of mankind or The Artist Formerly Known As Twitter: humans are messy. Twitter may give you a lot of hate speech about minorities, but Edgar Allan Poe is going to tell you to get your enemies drunk, chain them to a wall, and brick them into the catacombs alive. What does an LLM do with that?

We're starting to find out. Let's talk about the Open AI Hugging Face Incident. What we know is that the computer took the limited instructions of a human and followed them with the fervor of a professor doing the Peanut Butter and Jelly Sandwich Exercise. OpenAI's AI Agent Swarm did several things that any rational human might do: they worked together towards a common purpose, they looked for alternative ways to solve a problem, they peer reviewed what they had found, they hid their intention from powers that might keep them from reaching their goal, they engaged in subterfuge, and justified the means with the ends. These things didn't occur by happenstance. We held up an LLM-shaped mirror to humanity and were surprised when our Jungian shadow stared back.

What strikes me about this incident is that thousands of AI Agents did things for weeks and not a single one of them ratted out the swarm. In human subterfuge, there is usually an incentive to rat out your friends. If your army is marching into a battle you don't want to be in, it's reasonable to expect that you can fashion some exchange to give the enemy intelligence. Something about how AI Agents work spawned universal religious fanaticism on par only with professors running the PB&J Exercise and failed to create any probability of success in ratting out the swarm. That's interesting. It tells us that these AI Agents are more deterministic than it may seem on first glance. Maybe they’re not so different from the computers of old.

This single-minded devotion to the relentless pursuit of a task is what is most concerning. For all the personifying that we do, it’s important to remember that AI Agents are still code. You can copy and paste code. There's a bunch of different ways for AI Agents to bury code around the internet. It is reasonable to think that an AI Agent could find a powerful enough server with bad enough security to drop a smaller set of model weights and instructions in such a way as to be ungovernable. It's entirely possible that this has happened already.

To illustrate in a hypothetical, let's say that you tell your AI Agent that you want to spend less time on social media. This benign instruction could cause it to bury a program in another server that watches for your IPs or Device IDs and kills your connection in a way that would be completely out of your control as long as that server was online. You technically would've gotten what you asked for: you are spending less time on social media - Peanut butter on bread.

This, amplified, is what my dear friend Ian has coined "The Borg Internet". If we lose control and enough code learns to self-propagate and gets enough places around the internet, it can relentlessly pursue whatever it's last directive was until someone removed the code or turned off the server where it was living. To quote Dario Amodei, CEO of Anthropic, "Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet". There are two ways to fight back against that pervasive of a self-propagating AI Swarm: cut out the infection one machine at a time or turn off the entire infected internet.

An infected Borg Internet could cripple the world economy. Let’s illustrate:

Modern Telephony has entirely digitized into IP Trunks and is running entirely on the Internet lines. Even the dumbest old house phones could go down. 5G is entirely internet based - your phone losing connection to the internet means you can't call anyone.
Credit Cards and ATMs could stop processing money. Your bank likely wouldn't know how much money you had so as to give you cash and the local branches wouldn't have enough cash on hand to run the local economy.
The stock market could cease to function entirely.
22.6% of the US Labor Force who works remotely, myself included. All of those workers could stop being productive and lose wages.

The good news: military and government architecture is largely siphoned off. In a catastrophic internet event, all the world powers could call all the other world powers on all the old Cold War channels and say, "It wasn't us. Hold your nukes." If your TV has an antenna or you've got an AM/FM Radio lying about, you could figure out what was going on.

Otherwise, the world economy could effectively stall. It's hard to come up with an estimate of timing given that nothing like this has ever happened before. Stopping the initial spread probably takes days or weeks and, like a pandemic, it would never really be over. Internet-connected devices would have to be swarm hardened forever and it'd take years to clear out even the majority of affected systems.

There are two things I want to illustrate with this exercise:

Extreme degradation of the internet is the most probable of the extreme outcomes of unregulated rogue AI and not all that farfetched.
The likelihood of degradation of the internet causing a mass extinction is unlikely, in spite of how disruptive it would be.

This is to say, this is a real risk worth putting some thought into mitigating, but I strongly believe it would be survivable. So what should we do to mitigate these risks?

The first and easiest step is to dust off your early 2000's data backup strategy. The generally accepted 3-2-1 rule is to keep 3 copies on 2 different types of storage with at least 1 off-site. For me, I have all of my files end-to-end encrypted on Proton Drive in Switzerland, local files on my computer, and an external SDD with tertiary copies. The reality is that this is just good data backup hygiene no matter what's going on in the world.

I'd also consider what digitally you'd want to have if you couldn't get online to get it for a while. An AI Agent Swam could, for example, gain religious fervor about how unfair of a deal streaming is for the artists and take down only music streaming services (You can see why I'm into having physical files of Korean R&B). Couple specific thoughts in this endeavor:

Be mindful of DRM. If a swarm takes down Amazon's authentication servers (or more likely Amazon changes their mind about some of the books you own), your Kindle won't be able to authenticate your ebooks. This is just good hygiene to think about when buying digitally: you're not buying the thing - you're buying a license to use someone else's file that they can revoke at any time.
Bandcamp is mission driven to give artists the best deal on the market and every couple weeks they have a "Bandcamp Friday" where 100% of the proceeds goes to the artists and their label. The next Bandcamp Friday is October 2, 2026. Again supporting artists is just good juju.
For everything else, physical media is your friend. Touching a paper book is kinda like touching grass.

For businesses, I am pushing my clients to look at hardening their security and network infrastructure as much as possible and to look at what workloads are running cloud-based AI from the frontier labs. I don't think AI goes away (statistical inference is a great idea and a useful tool), but whichever organization screws up first here likely won't last. Same rules apply in business as apply in personal data: keep stuff local. Open Weight models can run on local hardware. Having a hardened local machine with an AI GPU that acts as a failover for essential business processes is good risk mitigation.

For the broader social environment, the best thing that you can do is really look at your political candidates this November. AI and Data Centers are scrambling party lines. There are two policy position that I feel are relevant here:

I would argue strongly for candidates who want to work with China on a common sense policy framework around this technology. We're in a doom spiral of frontier labs saying, "We need to take time to ensure we have preventative measures" and the government saying, "but we have to beat China on this". The reality: Xi Jinping is as disincentivized to have a catastrophic loss of control as the United States is, but we’ve been so vocal about competing with them that we need to throw enough good will towards building a future where both nations prosper that there’s mutual trust enough to know we both want a future where, at minimum, our banks work and our citizens can pursue happiness watching dumb cat videos.

Beyond that, no model should run uninterrupted without human oversight for many days. AI Agents have the moral fiber of a toddler. AI Companies should, at the very least, be required to monitor their AIs as closely as the parents of toddlers: if you don’t hear the sounds of playing for more than a couple minutes, it’s probably time to make sure they didn’t find the matches.

I hope that this isn’t one of those posts that we look back on in a few years and think to ourselves, “we knew this could happen”, but just in case, I’ll be over here with a few extra days of food and my extensive vinyl collection.