Hi Friends! Update 20 was more successful than we imagined and now we're working to build on that great foundation! Come take a look at the important features we have been working on this next update. Disclaimer: All of the content below is heavily work in progress and is subject to change.
Server Performance Enhancement
@Casey Hey guys, Casey here with a performance optimization update. Something I've been focusing on recently is optimizing the server performance of the game. We're always trying to figure out ways to get more players into a single server while keeping it running at high tick rate. In a game like Foxhole that has thousands of networked objects in a server with 140 players, understanding what exactly is costing us valuable milliseconds is fairly complicated.
The first big step to understanding this is being able to run a "profile", which is essentially a recording of what the server spends its time doing. Unreal Engine already has many systems built into it which can be used to profile a game in order to better understand what is taking the most CPU cycles and network bandwidth. However, in order for these profiles to be useful, they need to be done when the server is under a realistic load. Otherwise, you end up making performance optimizations that might make your profile results better, but not actually help when real players are playing.
The next big gotcha is that any profiling tool has some cost associated with it. It needs to record what the server is doing and write those stats and tracking data out to a file. This incurs overhead which in turn slows down the server itself. This limitation means we can't run these profiling tools on the live game with real players playing, or else we'd introduce additional lag and lower the maximum player count.
We needed a way to generate a realistic server load in our local development environment where we can run profiling tools on the server. This is where my bot army comes in!
What I've been working on is an army of bots that can all connect to the server on my development machine. These bots connect, join a faction, and then based on what test I'm running can either move around or stay still. Their behavior is currently very simple but has already yielded some really helpful results.
**Edit: These bots are just to test the server, bots are not coming to Foxhole**
The first challenge was actually being able to run enough clients at once to create a realistic server load. I knew I wanted at least 100 clients connected, to begin with, and ideally, I wanted to run them all on one machine to simplify starting and stopping tests. My first step was to create a client that didn't render at all or a "headless" client. This means that it takes no GPU resources at all, and lowers the CPU usage significantly. Next, I created an optimized assets file that skipped loading any rendering related data (like meshes) into memory, which cut the client footprint by half. At this point, I could run a single headless client that used about 800MB of RAM. To scale this to 100 on a single machine was still impractical.
Next, I used functionality in the engine normally used for split-screen gameplay to start up local players within each client. I was able to get 16 bot players per client instance after some tweaking. This meant I could run 8 clients, with 16 bot players each, at a total of about 6.4GB of RAM. On an 8 core machine, this means I could run 8 clients all locked to a single core. Once I was able to get all these clients connecting to the server, the next step was to load in a real save game from the live game and take measurements. I was able to identify several issues almost immediately once I was able to profile the server with all these clients connected and moving. An example of an optimization I did was disabling certain things on the character from being updated unnecessarily on the server. The player character in Foxhole has a bunch of sub-components attached to it (hands, legs, audio emitters, etc). Whenever the player moves, the game needs to update all the positions of those components. However, the only place these positions matter is when rendering the character or playing sounds, which the server doesn't do. In this case, it was a simple matter of disabling those components on the character only for the server. This ended saving us around 0.5ms of tick time, which is fairly significant!
This is just one example of the kind of optimizations that could only be seen with 128 or more players connected, with fewer players (or only one player) the amount of time spent doesn't add up enough to show up in the profile recording. I plan on expanding the bot's capabilities to drive vehicles, shoot, reload, and generally do more things that real players do. For now, I'm still working through a list of optimizations that came from our first round of testing. Hopefully, this was a good insight into the kind of work we do to allow Foxhole to support even more players at once!
Map Envy
If you've been following our blogs and streams since the new year began you'll know that one of our big goals for this year is to expand the world and make all regions feel more integrated. In Update 0.21, we're going to be taking the first steps towards this. The first big change you'll see is that all the island regions will now be connected to the mainland. Fisherman's Row will be connected Mooring County/Farranac Coast to the east, and Oarbreaker to the south. Oarbreaker will be connected to Farranac Coast/Westgate to the east, and Fisherman's Row to the north. Finally, Tempest Island will be connected to all 3 of the east coast island.
Next, the layouts of all three of these island regions will be redesigned with several goals in mind. First, we're making these regions more integrated with the mainland by making them better suited for supporting the mainland via water. The borders of these regions will be brought in closer to the islands and Ports have been positioned to make traversing to the mainland land a faster process. Second, within each region, we're making more areas of the map strategically relevant and were also adding more opportunity for beach landings and island hopping.
Finally, we're adding new strategic value to these islands. Salvage Mines, a variant of Scrap Mines will be placed in these regions and will be capable of gathering Upgrade Parts at a reliable rate. Holding these islands will be a key factor in building Facilities and high-end weapons like Battle Tanks in the war.
It's not the size of the boat, It's how you use it.
Since we are changing the layouts of the island regions to better integrate them into the rest of the mainland conflict, we needed a way for increased accessibility for movement over water. We have been aware of the constraints of barges, and how much of a strain it is to build one, so we decided that we needed to empower the average player should be able to move between islands more easily.
The Motorboat is the latest water vehicle we are working on Foxhole. As of this moment, it will allow for up to 3 soldiers (1 driver, and 2 passengers) to quickly move between beaches. Passengers will be able to equip their secondary items, similar to how a motorcycle works. The Motorboat is planned to be light on fuel consumption and Bmats while offering no protection to the soldiers inside. The current plan is to construct these vehicles on wet sand with a hammer.
As usual, these boats may seem overpowered/over-accessible at first, but our goal is to break the barrier to entry to traveling between islands. To alleviate any gameplay imbalance this may introduce to the rest of the world, we will restrict building these to island maps only for now
That wraps up another Dev Blog. Be sure to check out our Foxhole Dev Stream for more information about upcoming features. If you have any burning questions you want to be answered be sure to tweet them to @Matt directly on Discord or Twitter, if your question gets selected it will be answered Live on stream!
Great update!
Right now players have to mine resources themselves right?
Have you ever thought about instead having (very simple) NPCs "operate" (run around around and play some work/mine animations) the mines instead of having players do that tedious work. The players could instead guard the NPCs working in the mine, this would create a more interesting mini-game consisting of defense-building and counter-attacking around resource production than mining is right now.