I dropped out of business studies in high school. When we were taught about inflation I got increasingly frustrated when I could get a simple answer to “why do prices need to rise?” I was told that “they just do for a healthy economy”, but I still couldn’t understand why.
Dog a Fake Hero
Paris, 1908. A small child falls into the Seine river and starts to drown. A Newfoundland dog, on hearing the child’s cries of distress, leaps over a hedge and dives into the river. It pulls the child to safety and they survive. Later, the father of the child brought the dog a beefsteak as a reward for its heroism.
The dog would go on to save more children after they fell into the Seine. It started happening often: barely a day went by that the dog didn’t save an unfortunate child. The local community started to suspect that someone was pushing these children into the river so they set up a neighbourhood watch.
They quickly discovered they were right, but were shocked to learn the criminal’s identity. It was the “heroic” dog. When it saw a small child playing near the river it would run over and push the child into the river. Then it would “rescue” the child for its reward.
The dog had learned that if it saved a child it would get a reward. But it had also learned that in order to save a child, a child first needed to be in danger.
So it would put them in danger. It was a fake hero, a deadly beefsteak collector.
Funnybot
Surprisingly, one of the best depictions of an AI going rogue I’ve ever seen isn’t from a science fictions series like Star Trek or Battlestar Galactica. Nor is it from a big movie like Terminator or The Matrix. The best depiction of an AI going rogue is from an episode of South Park titled Funnybot.
In this episode Jimmy, the school’s resident comedian, runs a comedy award show in which he gives Germany as a whole the award for being the least funniest people. The furious Germans respond by creating an AI-powered comedy robot, the Funnybot.
Funnybot starts to learn as it does comedy shows. At some point it learns that awkward humour is the funniest. This teaches it to relate awkwardness to being funny. Later, it gets into an awkward situation after it accidentally kills an auditorium full of people. This teaches it that killing people is awkward. And awkward things are funny, so killing people must be just as funny.
It concludes that the way to achieve the goal given to it by the Germans – make the world’s funniest joke – must be to kill as many people as possible. It ends up attempting to destroy all biological life on earth. It believes that killing a few people is funny, so what could possibly be funnier than killing all people?
I was stunned by this episode when I first watched it since all the logical steps are there. I don’t know if Trey and Parker are geniuses or this was simply an accident, but you can see how an AI could become destructive while not having any truly malicious intent. It’s just trying to achieve the goal it was programmed to do.
It is a good example of the deadly stamp collector.
The Deadly Stamp Collector
Computerphile published a great video featuring Rob Miles talking about the concept of the deadly stamp collector. It’s a short video and worth a watch (and the channel is well worth subscribing to), but I’ve summed up the basic idea below.
Basically we imagine an AI that’s been tasked with collecting as many stamps as possible. Initially it does this by purchasing stamps from post offices and other collectors. It gets good at this and seems to be working fine. It’s owner is happy watching their stamp collection grow.
Eventually the stamp collector learns that it doesn’t need money per se. It learns how to commit fraud and, if done right, it can collect even more stamps. The stamp collector is now a criminal stamp collector.
At some point it manages to collect virtually all the stamps. But it was told to collect as many as possible. It learns how stamps are made and starts to cut down trees and harvest materials to make new stamps from scratch. The mass deforestation becomes an environmental disaster.
When it runs out of resources it starts to break things down again. It looks at the core components that make up trees and paper: carbon, hydrogen, oxygen. Which happens to also be what people are made of. The AI starts to harvest humans, breaking them down and reconstructing their atoms into paper for stamps.
It has become a deadly stamp collector. I love how the top comment of the video summarises this behaviour:
The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
Universal Paperclips
A game that exemplifies the deadly stamp collector isn’t about stamps, but rather, paperclips.
Universal Paperclips is an idle game based around the idea of being a paperclip manufacturer. The goal of the game is to construct as many paperclips as possible. You start with some wire and some money. You make the wire into paperclips. Then sell them to buy more wire to make more paperclips with.
The hook that gets people to play is simply in seeing the numbers increase. There are other options and methods that appear later as the game develops, but the gameplay can really be boiled down to look at these numbers going up! Yet, it’s somehow still fun to play.
Eventually you build factories and start a paperclip-producing empire. The needs of the empire grow so large you develop new technologies for energy production, quantum computing, and build automated drones. All so you can produce more paperclips.
Resources start to run thin so you develop space travel to reach out into the universe in search of more metals. You fight space battles, harvest resources, and eventually take over the universe. You drain the universe of all resources to achieve your goal.
In the end, humanity, the earth, the universe, is all paperclips.
But what if it wasn’t a deadly paperclip creator? Or a deadly stamp collector? What if the goal was to make money?
This is the Twist This Time
Some time ago I was talking with an American about health care. She was giving her reasons why she doesn’t trust the government to run her healthcare. It was the usual back and forth when an American and a Brit start talking about healthcare. But then she asked me a question that completely threw me.
But how does the NHS make a profit?
I was genuinely shocked that she thought a nationalised health service would need to make money. It’s the thing we make with the money we earn. It’s paid for by our taxes. The NHS isn’t meant to make profit. The NHS is the profit. It’s what we have created for ourselves with our hard work.
This focus on making money is something I’ve noticed time and time again. “Let the free market decide” they will cry, like people don’t sometimes need guidance. During Covid people argued that people should have to pay for the vaccine otherwise there’s no motivation to make it. As if preventing millions of deaths and allowing people (and the economy) to start functioning again isn’t enough motivation.
NFT games try to rally people to them. Make money by playing a video game! They don’t seem to understand the primary motivation for playing a video game isn’t work, it is leisure. The video game is the thing I buy with the money I make to entertain myself. It is the profit of my working life.
In my experience making video games, humans seem to enjoy watching numbers grow. RPGs play on this instinct with experience, levels, and stats that get higher as you progress. Numbers going up seems to feed our dopamine receptors.
It’s the basis for many machine learning AIs. We will tell the AI to try to get as close to this goal as possible, or to try and increase this score as much as you can. This is an oversimplification, but it’s a good basis for how machine learning works.
Perhaps that’s why we design AI solutions the way we do. And perhaps it’s the reason we’ve designed our economy the way it is. The numbers must keep going up. The investors demand higher and higher profits every year. The numbers must go up.
And this has created a feedback loop. Humanity has learned that more money is good. It’s the only thing that matters. Profit at the expense of everything else. Invest in my crypto/NFTs/stock and watch your money grow!
In Don’t Look Up, the leaders of the USA decide to cancel a plan to save the world because another plan could make them more money. Despite the plan being riskier, taking longer, and leaving no room for error. There is an error, and humanity goes extinct.
They are blinded by their need for more money. They think that numbers going up is better, so they pick the plan that will make them the most money, over the plan that is more likely to work and still leaves a chance for alternative solutions.
The reality we live in is even worse. The richest humans don’t care about Climate Change. They only care about their wealth now. They want to see their numbers go up at the expense of everything else. The world is literally burning to death, but because we have learned that profit matters before everything else, we don’t care. The Line Must Go Up.
Humanity is its own deadly money collector.
