Hey all, this Rick Stemm, writer/designer of Heroes Must Die. I'd like to share with you all a bit of detective work we did recently to squash a persistent bug. I don't know that any of this is out of the ordinary for bug testing, but there were so many false leads and obscuring factors having nothing to do with software or design that made this a really interesting case. So interesting that I relate it to you here, retelling the work of my friend and colleague Programmer Sam Marcus as we recline in our offices at 221B Baker Strret.
The Setup
For the uninitiated, Heroes Must Die combat works in two phases.
First is the Attack phase, wherein you have a limited time to plot a path through enemies, trying to build up a combo by matching type or element.
You then follow that path, hitting all enemies within, assuming you plotted a proper course without breaking the combo chain.
Then it switches to Defense, where enemies project attacks that you have to memorize, then dodge to a safe spot.
The Problem
For months and months we would get this occasional problem where the two modes would overlap, Offense and Defense running simultaneously in this mess of impossible overlapping actions. It was not pretty.
Not something you want in your game as you prep for Steam release. This bug kept popping up intermittently for months. We had two main issues:
1. Anyone who saw it had no idea what caused it.
2. Sam (programmer) and I could NOT reproduce it.
3. We had just cleaned up the combat code so any obvious problems were already taken care of.
The Clues
We had a little bit of information to go on, however.
1. It happened to other people but not us, so there was possibly something about the devs or our builds that was different.
2. It happened by far most often in a particular level against a particular faction (Fighting the Network of Masks faux-superheroes in Gothic City). That was the only place with any consistent errors, so there was possible something about that level or those combatants.
We did what you would expect:
Sam checked the code all throughout combat, and came up with nothing.
I checked the specific level, combats, and combatant prefabs to see if there was anything wonky, and came up with nothing.
We tried like hell to reproduce or get info from people who saw it, but came up with nothing. It was too rare and sudden to have any clues when it happened.
Nothing!!
The Breakthrough
Twitch Streamer whitejak (who it turns out is a friend of a friend, though that's now how he saw the game) did an awesome livestream of HMD which I stopped by to do some dev commentary.
Lo and behold, the bug popped up. Multiple times! I was distraught that it was still happening but Sam was thrilled (I could tell because he was ever s slightly less sarcastic than normal). He reminded that now we had video evidence and could use that to solve. Jak sent us the link to the broadcast and I delved in to see what I could see, now that we actually had repeated evidence of it happening.
The Game's Afoot
The bug came up three times during the broadcast. After isolating the incidents where it happened, I looked for commonalities. Some we already knew, which had led us to dead-ends (it happened in the usual level against the usual faction). Fairly quickly I noticed another common thread - the bug always happened when time ran out in Attack mode.
So I went and tested for a while, running those same combats in that level, taking a variety of actions but always letting time run out. Still Nothing!
So I went back to the video for more clues. Though as a writer I am a noted Sherlockian no pipe-chewing, violin-playing, or Watson-shaming helped. I just stared at the screen over and over looking for the smallest detail. And small it was - I eventually noticed Jak's mouse move off to the corner of the screen each time. It was obscured by the Twitch interface, but could he be hitting the Attack button?
Eureka. I had found it. The bug occurred when you hit the Attack button at the same time the timer ran out. This would double-trigger the end of phase and cause the overlap. So obvious in retrospect.
Why Did We Miss It?
The most interesting part of this whole experience to me, and to other game devs I think, is the variety of red herrings.
1. Why Couldn't We Reproduce
Simple - we were too good at our own game. We either had our paths set with plenty of time left, or knew that it auto-committed your route when time ran out, so we would just let it go and use that last half second to keep planning in tight situations. It would never occur to us to rush to click Attack at the last second.
2. Why Did it Appear in That Level with Those Enemies
Gothic City is the third major level in the game. There is a prologue, then a level where you get comfortable with mechanics, then this one. So essentially, it's the level where the game starts to get hard. Before then it is teaching you, and after then you are more comfortable. But at this point, the patterns on Offense mode are, for the first time, hard to see. Thus you are more likely to scramble against time trying to find them.
3. Why Could No One Report
It happened when trying to click Attack as time ran out, perhaps the most stressful or focused part of gameplay. Of course no one would remember details of what happened when they were racing to beat the clock and survive.
Bug Fixed, Lessons Learned
Sam fixed it in no time, but we both found the experience extremely interesting and enlightening. An example of false perceptions and coincidences, making a mystery unsolvable by someone too close to it. Thankfully we lucked into the video evidence, did our detective work, and can move full Steam ahead.
