Elevated Aiming Bug
I recently worked on an aiming bug, and the fix was pretty interesting so I thought it might warrant a quick breakdown in the devblog.
The original bug report was that players were having trouble aiming uphill while taking cover behind a sandbag, as seen in the screenshot:
The first step when debugging any kind of problem like this is to visualize it, to see what’s really going on. Using the trace visualization tools, I found the reason the aim line was getting stuck on the sandbag was due to what we call the “muzzle blocked” check. This trace is there to prevent the ability to shoot when the gun muzzle is inside some geometry. Here is an example of what that trace visualization looks like, with the character moving so it’s easier to see what’s going on:
After this I added some visualization to show where the muzzle location was. Since it would be prohibitively expensive to run full skeletal animation for each character on the server, in Foxhole the muzzle positions are modeled as a fixed offset (for each stance) from the character’s capsule. This means that the muzzle location, which is where hit traces and the actual bullets come from, may not always match up exactly with the visuals. This difference is especially evident with short-barreled weapons like the pistol.
The screenshot below shows the true muzzle position (blue cross) and the orange line represents the difference between that and the visual muzzle on the weapon.
There was actually a bug here which was that this muzzle position component was a bit too low, causing the muzzle checks to fail when aiming over a sandbag, because the muzzle position was actually below the top. I fixed that issue, and also added some debug draw to show where the shot on the server occurred. This is represented in the screenshots below as a red or green line, depending on if the trace hit something on the server.
In this screenshot you can see a shot taken when backed away from the sandbag while crouched, and below it a shot taken when close enough to the sandbag to trigger the bug. The short green line indicates the shot actually was still hitting the back of the sandbag because of the muzzle obstruction check mentioned above.
I investigated further, and find that other aiming scenarios suffered from the same problem, but in a different way. The screenshot below shows two cases of shooting down from a height, where the server shot and the client aim don’t match up properly:
Essentially, this bug was caused by the logical representation of the weapon muzzle not matching the visuals the player sees when aiming. Now that the main problem has been established, on to the actual fix!
In this case, the server needs additional information about where the gun muzzle (as seen visually) is. However, we can’t let the client simply send the muzzle location for where the shot should originate. If the server trusts the client to specify this information, then aim hacks could be created that simply set the muzzle as next to an enemy, for example.
That being said, if we instead allow the client to specify an offset from where the logical muzzle location is, and constrain the size of that offset server-side, we can get the best of both worlds. We can preserve the performance saving optimization of not running the animations on the server, and also allow the shot to match the muzzle. The server will apply the client-specified muzzle offset, but only if it’s within a very small margin of the logical muzzle as simulated on the server. All the same muzzle block checks still happen server side, so this offset can’t be abused to say, shoot through a thin wall.
Here are some screenshots of the same scenarios as seen above, but with the fix applied:
The tough part about fixing this one was to make a fix that preserved both server performance, and didn’t open up the server to trusting the client too much. I think I achieved that here and I’m happy with the results!
Hopefully this was an interesting insight into how a typically bug fix might look.
A bit of a history lesson for ya. Godcrofts used to be an island on the original Fisherman’s Row. Crazy, I know. We took Godscroft and just jammed it into the mainland on the opposite coast, why? Because while planning for the world revamp, we had to make 9 new maps in a very short time and by not needing to make a new island map we reduced that number to 8.
That’s it, really. That’s not to say we wanted to compromise on the world quality, on the contrary. We wanted to find as much extra time as possible to spend on polish for the mainland maps where we knew the majority of players would be.
But poor Godscroft. It didn’t have as many towns as its mirror, Fisherman’s Row. Huge swathes of land were completely featureless after we gutted the port bases. There were roads that led nowhere. In short, it was little more than a placeholder until we had more time to clean it up.
Well we cleaned it up. Godscroft is predominantly composed of two major islands that shield a strategically critical inlet. The map connects to both Tempest Island and Endless shore, providing alternate routes for launching an invasion into those maps. We hope it sees some action this patch.
Foxhole Maps have changed a lot of the years. Many of you probably remember the combat prototype map and have seen the changes (good and bad) on the map screen. Game develop is usually a balancing act between what is possible, and what is scalable/repeatable. There was a large part of Foxhole’s development where I would manually craft every single map image. This is fine when there are 1-4 maps, but as a game grows especially one that changes as often as Foxhole does, this manual generation was not sustainable. My process looked something like this:
Open a single map and wait anywhere from 2-10 minutes for the map assets to load. The image below shows a map loading in realtime.
I would then take a screenshot within an in-game camera we set up, and export that screenshot into Photoshop. Every map was a different resolution meaning I needed a separate PSD (Photoshop Document) file per region.
Begin the painfully slow process of selecting and masking individual elements of the map and filling the mask with the appropriate color and texture. A single map would take anywhere from about 5 - 60 minutes to generate.
Trace over every visible road by hand.
Repeat for every map in the game
Around Update 25 we had around 16 in game maps, so I would essentially have to take an entire workday to generate them. This was such a bottleneck in our pipeline that we ended up having to give myself less modeling/artwork so that I could generate the maps at the end of a development cycle. When Matt or Mark would come by my desk and inform me that every map needed to be generated for the build, I would not so subtly show my frustration to the entire office.
The maps would go through dev branch and players would inevitably find errors and bugs, the maps would be fixed, and depending on the severity of the issue I would have to generate the map image entirely again, especially if a border was moved. The gif below roughly shows the process.
With the somewhat recent decision to turn our regions into hexagon shapes, we seriously had to take a look at Map Generation, and a way to automate it. There was no feasible way that I could generate an accurate map image for 23 maps. Not only did the borders have to line up, but we also planned to work very quickly and iterating often.
How did Hexagon regions change the way we work? What efficiencies did we develop? I have compiled a list of heavily oversimplified steps in order for you to understand the process. It goes without saying that all of these steps required many, many hours of research and development to find a solution that suited our needs.
Hexagons standardized the map resolution at 2048x1776. A single PSD file now housed every map in the game.
We have a floating water plane mesh that is invisible to players that changes the colour of the water to stand out from the land
We automatically generate a screenshot every time a level artist saves a map, meaning that every image contains the latest, most up to date information
We could now batch export every map into a singular directory on our work computers.
In Photoshop we can point to that directory using a tool called ‘Smart Objects’ automatically updating the contents of our photoshop file
We use Gradient Maps and other filters where we assign specific colours to the greyscale elements of the map screenshots, as well as clean up the noise
We can batch export every map at once, and batch import every map into the engine.
Maps are automatically stitched together in engine
My vacation time and the end of an update cycle can now be decoupled
These changes have increased our efficiency by an order of magnitude, allowing us to not spend resources manually generating maps. An artist on our team can edit every map simultaneous in a few clicks, instead of manually drawing each one. These generated maps are not a perfect solution unfortunately and have its drawbacks. Roads and particularly cities are harder to see. Rocks on grass patches are nearly invisible. The resolution quality is lower and less crisp. We have seen all of the issues and we are looking into ways of solving the issues. Matt and I have been exploring a solution to the rock visibility issue by essentially assigning a bright flat colour to every rock when the Camera is a certain distance away. This is a system called Level of Detail (LOD) and many games use this to increase performance by making distant objects ‘less expensive’ on your computer the further they are for the camera. Since we are a top-down game, we don’t really take advantage of this often, but perhaps this can be a solution that helps with other map-related issues. I will have to tweak the parameters of gradient map to factor in the new rock grey scale values, but potentially this could help!
XOXO — Mr.Clapfoot.
That wraps up another Devblog. Be sure to check out our Foxhole Devstream for more information about upcoming features. If you have any burning questions you want answered be sure to tweet them to @Matt directly on Discord or Twitter, if your question gets selected it will be answered live on stream!