As a developer, the one thing you want people to do is to be able to play your game when you launch it. Unfortunately, that wasn’t possible for many players after the launch of one title I worked on. The PC version of the game was crashing on launch, and I was tasked to investigate.
On this project, launch day was almost a big success. The console version of the game was running well, and we were seeing a lot of PC players as well. But we kept getting reports from some players that the game would crash as soon as it launched. This was happening to a significant portion of players on PC, enough that it was all over the internet.
Finding the Crash
To figure out what was going on, we needed to find something that was common among the crash reports. We collect the specs of people’s computers when they report a crash, so we just needed to look through these to see if anything stood out. One of our QA was tasked with this, and they noticed that all the crashing machines were using multicore processors.
Multicore processors were something that were only just becoming popular. At the time most people would still be using single cores, but PC gamers like to have more powerful machines. So a larger proportion of gamers had multicore machines and experienced the crash. Unfortunately, most of our machines were still single core, so QA had to quickly requisition a machine in order to test for the crash.
They got their machine, and managed to reproduce the bug as soon as they started the game. Since they were running in debug they were able to generate a crash dump with a stack trace. This is where I enter the story. They sent it over to me so I could take a look.
Looking at the stack, I could see that it was crashing in our custom anticheat code. It was designed to crash the game if it detected that a player was cheating. For some reason it was detecting everyone using a multicore processor as a cheater, and that’s why the game was crashing.
So the next question was obvious. Why did the anticheat code think that anyone using a multicore was cheating? To explain this we need to take a slight detour into video game history.
Overclocking (and Underclocking)
In the early days of computers, most processors ran at the same speed. Games were written to just run at the fastest speed. This was fine at first. While different processors would run at slightly different speeds, games would still run at a steady pace.
As processors started becoming more powerful, these games started running faster. So fast, in fact, that they were unplayable. In order to get older games like X-Com to run on newer computers, you had to not only restart the computer in DOS mode, but also underclock (slow down) the processor so that the game would run at a playable speed.
Games started getting better at this. They would use the internal clock to time each frame, but they would still run as fast as they could. You might get 15 FPS (frames-per-second) on one machine and 30 FPS on the other, but overall the games events would still run at the same speed.
Unfortunately games still didn’t get things quite right. Sometimes you could get an advantage with a faster frame rate. In early games your (e.g.) movement would be tied to the framerate rather than milliseconds passed, so running the game faster would let you move faster in online games. This was a common way to cheat in first-person shooter games.
First-person shooter games like the one we had just released. And this brings us back to our crash bug. The anticheat code where the game was crashing was attempting to detect a processor that had been overclocked.
The Anticheat Code
The way that the anticheat was attempting to detect overclocking was a rather simple solution. When a core is running multiple applications, it splits its processing time between them. It will process Application A for a bit, then Application B, then back to Application A, back to Application B, and so on. With more applications it would cycle them the same way (A-B-C-A-B-C-A…).
In order to detect an overclocked PC, we needed to know how fast the game was running. We knew the minimum amount of time a single frame would take, so we needed a way to measure this time. To do this we would launch a second application that would run for a short amount of time, then close itself.
By detecting how long this application would take to shut down, we would know how long our game was taking to process a single frame. If this time was below a certain threshold, the computer was assumed to be overclocked: we had a cheater on our hands.
False Positive
This worked perfectly for single core processors. The problem was that something slightly different happens if you have multiple cores. In a single core processor, the computer can only process one program at once. This is why it will split the processing time between applications.
But in a multicore processor, the computer is able to process multiple applications at the same time. If you have 2 or more cores than you can run 2 applications in parallel without either one impacting the speed of the other.
The problem with our anticheat code was that when you ran it on a multicore processor, the test application would launch on a different core. This meant the speed of the process wasn’t affected by the game application, and it would finish almost instantly. When this happened the timer would detect this and report an overclocked processor.
Then the game would crash.
The Fix
The quick fix was to release a quick patch that disabled this part of the anticheat code. As soon as we did this, people were able to play the game they purchased.
Anticheat is talked about a lot these days, since online games often have cheaters looking for an unfair advantage. Many might assume that developers aren’t working hard enough to prevent cheaters, but this really isn’t the case.
The truth is that detecting and preventing cheaters is really really hard to do. It’s very easy to get a false positive and ban someone unfairly, so you need to work really hard to ensure that doesn’t happen. There’s a reason that there are now companies that specialise in anticheat software. Being able to shift a lot of the work that goes into this very difficult problem to another company enables your team to focus on developing other parts of the game.
Moderation tools and reporting works well at avoiding unfair bans, but these require manpower and if a game has millions of players, it can be nigh but impossible to get a team that could react to every cheater in a reasonable time. And if a player doesn’t care about being banned, there’s nothing to stop them cheating until they get caught.
Really there isn’t a 100% solution. There will always be a push and pull between cheaters and game developers. All developers can do is continue to improve their anticheat tools, and hope that they eliminate enough.
