Good news everyone! The performance problems are solved (for now). In essence there have been two major problems:
If you follow our dev blog you might already know that we are using an actor based architecture in the backend. Every actor, for example a company, has multiple behaviors, each responsible for a different part of the actor. Examples are the accounting behavior, fleet behavior, base behavior and so on. Actors store their state as events in the database and when they wake up they go through all of their events and recreate the state of their behaviors.
Since this test was never intended to run this long we didn’t bother to optimize the behaviors memory-wise. Up until recently each behavior would just keep all the data that was being read from the database. For example: if a company had thousands of notifications accumulated over time, all of them were kept in memory, even though they are just dead weight.
We quickly realized that this could be a major factor of the performance issue and tried to solve this. Instead of going through hundreds of behaviors and try to figure out which of them could accumulate dead weight, we tried to do that programmatically by measuring the behavior’s memory footprint. I tried several methods to do this, but it turned out to be more difficult than I anticipated. Most of the solutions just made the problem worse by an order of magnitude. Eventually we resorted to the manual way: we identified the three most promising behaviors, implemented means to get rid of unused data and that reduced the overall memory footprint by more than a third.
The second problem was a bit trickier to detect, since it didn’t leave any trails in our logs. When a company liquidates, all behaviors receive a callback, basically asking them to clean up. The local market behavior for example deletes all LM ads and once this is done the COLIQ can continue. To delete a LM ad a transaction with the local market actor is opened and in this particular case it failed, since the local market actor could not find the ad that should be deleted. This caused an endless loop since the local market behavior on the company side tried again and again. To make it even worse, transactions store events in the database, so this behavior was writing huge amounts of events to the database, consuming memory and of course lots of CPU time.
With these problems solved now I finally can resume my work on the Populous release :)
This week was spent planning for new content and research on who the “perfect Prosperous Universe adopter” is. For both SEO and website design, I will be creating a Release Roadmap to showcase where Prosperous Universe came from, where it is now, and a glimpse at what is to come. I think this will be a really cool asset to have and share with the core audience. It has involved going through past devlogs and forum posts to see which Alphas were important and what features were added. It has given me a unique perspective on what the game was like in the past and how far along it has come.
Another thing I have been working on is called the HALO Strategy, which gives us insight into the hopes, dreams, fears, and barriers for our “dream adopter.” Although Prosperous Universe is for everyone, there are certain qualities and preferences that many of our players share, which is what marketing is all about: connecting to the right people. It is interesting delving into the player psychology and taking a look at what parts of the game resonate best with our audience. I am also thinking about contacting some players for interviews. Reach out to me if you would like to participate!
Subscribe via RSS