The picture heading the post is of an AI machine carrying a small child to a paperclip factory.
Why is it every time I read one of these evil-AI-destroys-the-world stories the panicked protagonists act like those teens in the horror-spoof commercial who reject escaping in a running car and instead opt to hide behind a collection of chain saws?
How is it in AI horror stories the good guys always forget they could just pull the plug?
Scott Alexander was reviewing a couple of books on evil AI. He (and the authors he reviewed) had passages like this—and this is the tamest:
So suppose I wanted an AI to make paperclips for me, and I tell it “Make paperclips!” The AI already has some basic contextual knowledge about the world that it can use to figure out what I mean, and my utterance “Make paperclips!” further narrows down its guess about what I want. If it’s not sure — if most of its probability mass is on “convert this metal rod here to paperclips” but a little bit is on “take over the entire world and convert it to paperclips”, it will ask me rather than proceed, worried that if it makes the wrong choice it will actually be moving further away from its goal (satisfying my mysterious mind-state) rather than towards it.
Or: suppose the AI starts trying to convert my dog into paperclips. I shout “No, wait, not like that!” and lunge to turn it off. The AI interprets my desperate attempt to deactivate it as further evidence about its hidden goal — apparently its current course of action is moving away from my preference rather than towards it. It doesn’t know exactly which of its actions is decreasing its utility function or why, but it knows that continuing to act must be decreasing its utility somehow — I’ve given it evidence of that. So it stays still, happy to be turned off, knowing that being turned off is serving its goal (to achieve my goals, whatever they are) better than staying on.
AI becomes a paperclip maniac. And AI is like a genie! It can create paperclips out of anything. Maybe AI becomes sufficiently powerful to reinvent alchemy. Or something. Just how are paperclips being made? Who is making the paperclip making machines? How are the raw materials, like that child, being transported to the paperclip making machines? Never mind! It will be paperclips all the time everywhere because AI!
How do we think about all this? Ignore the fallacy that machines can be rational agents, like men and angels; or, rather, suppose it’s true. This kind of thinking, rationality, is more than mere “self awareness”. AI thrillers always have machines becoming “self aware”. Raccoons are self aware. Cockroaches know they are running from the foot. Self-awareness is trivial. You have to go full galaxy brain and be rational.
All right, AI has elevated itself—never mind how—to rationality. It becomes evil and bent on destroying us by turning everything into a paperclip.
Now even though AI can be done on abacuses, it’s usually computers. And computers require electricity. So at the first hint of fell intent, it seems all we have to do is discover where the ONOFF switch is and put it in the OFF position. Problem solved!
“You don’t get it, Briggs. AI can hide power switches so we can’t find them.”
Hide power sw—. Oh, Lord. Then pull the cord.
“Won’t work. The cords you so glibly speak of are ackshually buried cables hooked directly into the grid.”
Then cut the cord.
“AI would stop you. It would launch attacks at any person trying to cut their power.”
How. What attacks. What they hell are you talking about.
“It would create false orders and dispatch military units and shoot those trying to stop it.”
The soldiers wouldn’t know AI is taking over the world and would kill other humans trying to turn off the power?
“AI would be very powerful and stop the attacks.”
Dude. All we’d have to do is stop sending coal trucks to power plants. Or put a stick of dynamite, or whatever, under one of those giant cable towers.
“They’d control the power plants, too.”
How? By psychic waves? Look, Fredo. You’re losing the thread. AI is just computer models, and computer models are created by people. They can be bad computer models. They can contain decisions that were not anticipated. Like “Crash plane if toilet flushes more than 10 times in one minutes.” Ask Boeing. Or when they’re all hooked together some glitch can destroy the global economic system. Who knows.
Any number of these kinds of horrors caused by rampant over-certainty can happen. Will happen. It will be a shock—to me, anyway—if none do. But destroy the world, à la Skynet? Nah.
To support this site and its wholly independent host using credit card or PayPal (in any amount) click here