• Register

Currently developing GearBlocks. I guess you could say I'm an industry veteran (in other words, I'm old ;)). After having worked as a programmer in "AAA" games for around 15 years, I decided to try an experiment, could I develop and release my own game? Jury's still out on that, but things are going pretty well so far.

RSS My Blogs

Well, seems like another two months has gone by in a flash! I was away visiting family for some of this time, and the work I've been doing on the game has resulted in a frustrating lack of progress to show for it, but here's an update on what I've been up to anyway.

Saved game serialization

Up until now, saved games and constructions have been serialized via a binary stream, with no formatting, just using BinaryReader and BinaryWriter directly. This is fast and results in a compact file size, but has one huge disadvantage, a lack of version tolerance. In other words, if I add or remove any variables to be saved, or reorder them, then old saved game files will no longer load correctly. To work around this I wrote code to check a version number in the saved file, and then convert things over for each added or removed variable. This is a hack really, and has resulted in rather messy and hard to maintain code.

This situation is bad enough with just the demo version of the game out there, with a cut down feature set. Maintaining saved game backwards compatibility will only get harder once the full version is released.

Ideally, I need a properly structured save file format, with some kind of key value pairing that would allow for version tolerance, but wouldn't bloat the file size too much.

BinaryFormatter

First I investigated using BinaryFormatter, because it allows for version tolerance via optional fields, but I couldn't get it to work when deserializing MonoBehaviour classes. I need to be able to instantiate the MonoBehaviour and then populate serialized values into it, not have the deserialization process itself try and allocate a new MonoBehaviour (which is not allowed by Unity). I thought maybe using a serialization surrogate might allow for this, but couldn't figure out a way to make it work. The other downside of BinaryFormatter is all the assembly and type data it saves out adds quite a bit to the file size.

Json serialization

So after looking around for other possible solutions, I decided to try Json. This provides the key value pairs I need in a fairly compact structured format. I used Json.NET from Newtonsoft (provided via a Unity asset store package for ease of integration) which seemed really promising, it's very easy to use and highly configurable. In most cases there's very little additional code to write, you can just use the JsonProperty attribute to specify which class properties to serialize, and configure how they're serialized. Also, it allows for populating properties of a MonoBehaviour that has already been allocated, by using JsonSerializer.Populate() inside a JsonConverter.

Still, it took me several weeks to get Json serialization working for both saved constructions and full saved games, there were a few stumbling blocks along the way which took time to work around, as did just learning how to best use the Json.NET API. The end result seemed great though, it solved the version tolerance problem, and the code was so much simpler and cleaner.

One issue was that the resulting file sizes of the text based Json format were huge. Given that the game uses the same serialization code path to send construction data between networked players, this was a problem. So, I switched over to using Bson (the binary equivalent to Json), and also compressed the data via a DeflateStream. This worked well, the resulting file sizes actually ending up smaller than my original binary stream format.

Performance and memory problems

At this point I thought I was good to go, but then I started profiling the Json serialization with large saved game files (more than a thousand parts), and realized I was in trouble. Firstly, deserializing the saved game was more than twice as slow using Json vs. the old binary stream method. This wasn't a complete disaster as the load times weren't terribly long in the first place. The other more serious issue was that the Json deserialization did an enormous number of tiny GC allocations (as in millions of allocs, totalling hundreds of MB!).

I found that reducing the JsonProperty name lengths helped slightly with this but not to any significant degree. I spent quite a lot of time restructuring the code that loads the various modules in the game (player, constructions, time of day, etc.) to try and deserialize from the file stream more efficiently, but this made very little difference to performance or memory usage unfortunately (the resulting code was cleaner than before though so I guess the refactoring was worth doing anyway).

I'm annoyed with myself that I didn't do enough tests to pick up on these problems before putting all the effort in to convert the game over to use Json. If I'd known ahead of time, I probably wouldn't have bothered.

So now I'm not sure what to do. If I stick with the old binary stream solution, then all the Json serialization effort will have been wasted and I'm still stuck with difficult to maintain code for backwards compatibility. But the Json serialization option as it stands isn't acceptable, I'd need to do something to resolve the memory and performance issues. One possibility would be to manually serialize everything (i.e. use JsonReader / JsonWriter directly rather than JsonSerializer), supposedly this is the fastest way as it avoids overhead from reflection and so on.

I've decided for now to put all this to one side, think about it some more, and come back to it later. In the meantime I really need to get back to trying to make some positive progress with the rest of the game!

Player animations

Player animations

dangersam Blog
Body parts

Body parts

dangersam Blog
Start a group Groups
Indie Devs

Indie Devs

1,658 members Hobbies & Interests

A group dedicated to indie and standalone game development.

Unity Games

Unity Games

1,812 members Hobbies & Interests

For all Unity developers and developers-to-be, both beginners and professionals!

Blenderheads

Blenderheads

572 members Arts & Literature

For us folks who like to stay Free and use the Blender 3d program over every other costly options!

SmashHammer Games

SmashHammer Games

1 member Developer & Publisher

Indie developer of GearBlocks - a game about creative building, interactive physics based machines, and gears - lots of gears.

Comments
clashka43
clashka43

I really would like to add something to the game. For example, new part) Just how to do it, I don't know(

Reply Good karma Bad karma+1 vote
clashka43
clashka43

Hi sam! I very like you game)

Reply Good karma Bad karma+1 vote
Post a comment
Sign in or join with:

Only registered members can share their thoughts. So come on! Join the community today (totally free - or sign in with your social account on the right) and join in the conversation.

Level
Avatar
Avatar
Last Online
Country
Canada 🇨🇦
Gender
Male
Friends
Become friends
Member watch
Follow
Statistics
Rank
6,871 of 697,067
Visitors
8,001 (80 today)
Time Online
2 days
Activity Points
498
Watchers
2 members
Comments
110
Tags
3
Site visits
2,847
Contact
Contact
Send Message
Twitter

Latest tweets from @dangersamn

Saved game serialization - Well, seems like another two months has gone by in a flash!  I was away visiting family… T.co

Oct 6 2019

Player animations - The janky placeholder player animations that have been in the game for ages were long overdue f… T.co

Aug 3 2019

Body parts - Time for a long overdue update!  I’ve done a lot over the past couple of months so I’ll have to split… T.co

Aug 1 2019

Text decals, data links, and electronics - I’ve been wiped out with a nasty bug over the last couple of weeks which… T.co

Jun 3 2019

Linker tool, power, and more - Here’s a quick update on what I’ve been working on over the last month or... Tmblr.co

May 12 2019

Better building for multi-player - Most of my time over the past few weeks has been spent on a major... Tmblr.co

Mar 20 2019

Material tool test in #GearBlocks Youtu.be #gamedev #indiedev #madewithunity

Feb 13 2019

ScriptableObjects For Fun and Profit - Well, it’s been a while, so time for a progress update I think!... Tmblr.co

Feb 12 2019

Friends
uberblah Online