Announcing Dæmon engine Beta 0.52 news

We merged our 0.52 branch and tagged the 0.52 release. Unvanquished 0.52 Beta will be released on 14th of May, this Friday. The countdown is ticking! While we are packaging the game and reaching out to server owners to update their servers to be ready on that day, we announce the Dæmon engine.

The Dæmon engine is a raw engine, the software component executing the game code in a virtual machine and rendering the game while dealing with input and network. This is a component of an open source ecosystem but not an integrated editing platform like Godot.

Let’s introduce the Dæmon engine

Unvanquished and Dæmon engine history.

The Dæmon game engine is tailored for fast paced arena games.

Initially based on the id Tech 3 engine due to Tremulous lineage, and inheriting both improvements from the Wolfenstein: Enemy Territory branch and the Xreal Engine, the Dæmon engine evolved to support modern techniques like a tiled dynamic light renderer, bone-based model animation including Doom3’s MD5Mesh and Inter Quake Model “IQM” support, surface multitexturing including deluxe mapping, normal and relief mapping, the good old Phong lighting model and (new in town!) PBR materials, while keeping the proven BSP technique for rendering the world. Special effects like bloom, rim lighting, motion blur, heat haze, and color grading are supported.

You may have noticed some ongoing trend for some classic game engine mechanisms on the market, even commercial games like Ion Fury being based on EDuke32 (Build engine derivative, Duke Nukem lineage) or Wrath: Aeon of Ruin based on DarkPlaces (Quake lineage), both being free and open source engines like Dæmon.

So if you’re looking for a game engine with modern rendering techniques and model animation while keeping some classic game experience, the Dæmon engine is for you! Not being as old-school as something based on EDuke32 or classic Doom with sprite-based characters (though you can make games with similar feelings with it), the Dæmon engine positions itself more in that family of games like Unreal, Quake or Doom 3 with somewhat realistic environments but with nervous gameplay and no “return to combat area” message.

The Dæmon engine lives as a dedicated open source project, with development currently hosted on GitHub. If you want to base your game on this engine, you’re welcome, it’s made for!

XreaL then Dæmon were always a good place to showcase new techniques. After improvements on the rendering side in the XreaL era, at Unvanquished time we migrated from legacy QVM to NativeClient (now WebAssembly is in the pipe), then migrated bot code from the legacy AAS system to navigation meshes, and we are now back at improving rendering.

The Dæmon engine and the Unvanquished game are welcoming places for experiments. If you look for an open source engine to implement this or that new rendering technique or looking for an open source game to try some advanced AI thing for bots or anything fancy, hey, maybe you found the project you need!

Hardware and system support

Unvanquished GPU Compatibility matrix.

The Dæmon engine and the Unvanquished game itself are tested on a very wide range of hardware and software configurations (see the Unvanquished GPU compatibility matrix). For the 0.52 release, we tested more than 60 graphics cards on about 80 configurations.

Unvanquished itself requires cards starting with OpenGL 3. The game engine is able to run properly on GPUs starting with OpenGL 2.1, including AGP cards from July of 2002. Yes, we validated PCI Express, AGP and even PCI cards.

We improved a special workaround for a buggy Nvidia driver. In the past some people reported problems with Nvidia hardware. We did more tests and it appears the bug occurs with GT218-based Nvidia GPUs running the 340 driver. The 340 driver is the latest driver for this range of GPUs and because Nvidia driver is proprietary, no one will never fix it. This faulty driver erroneously reports supporting a feature it does not support, so to enable or disable this feature the game cannot trust what the driver says and we have to do some guessing to detect the faulty driver instead, so we disable the feature used by the optimization when this driver is detected.

We worked hard to disable unused code when features are disabled. For example we made sure to not load normal maps and to not compile the normal mapping code in GLSL shaders when normal mapping is disabled. Doing that not only helped to improve shaders compilation time and GPU memory usage on less performant systems, it also allowed some very old systems to just work. We discovered the GLSL code of the tiled renderer does too complex of operations for the tiny ALU of some older GPUs, breaking the rendering. Because such hardware is too old to sustain dynamic lighting anyway and this feature should be disabled by the user, making sure this GLSL code is not compiled when this feature is disabled just enabled those old cards to run the game. And that’s how the Radeon 9700 Pro from July 2002 works with the engine: ~40 fps if you ignore the specific Unvanquished model case.

Although Dæmon engine’s primary purpose is not to be a retro engine to run on obsolete hardware, but to implement modern techniques like tiled rendering, relief mapping or PBR (see below), we don’t leave behind players with slow hardware.

As said, the Dæmon engine supports OpenGL 2.1, but Unvanquished requires OpenGL 3. The reason is that we use animated models with too many bones to get them accelerated on pre-OpenGL 3 hardware. If you plan to make a game with the Dæmon engine, you know you may go back to 18 years old graphic cards while keeping the game playable just by using simpler models (up to 41 bones).

Of course modern GPUs are the best way to enjoy the Dæmon engine, for example we tested GCN and RDNA ones with the same care. For example the AMD R9 390X from 2015 can handle ultra preset with 4K resolution at 144Hz. The AMD R7 Embedded in R series APU can handle 2K resolution with medium preset as well as Intel UHD.

We not only verified it worked, we fixed the related bugs in order to make it works on that wide range of hardware, even working around defective Nvidia drivers we cannot fix ourselves. This may make the Dæmon engine the most widely tested open source id Tech derivative on that topic, as far as we know.

Doing those tests also uncovered a pile of problems with the Linux kernel PCI code affecting both PCI, AGP and PCI Express GPUs, this reminds how precious this kind of testing is, not only for gaming.

The Dæmon engine builds and runs on Linux, Windows and macOS. We provide some tooling (like a Docker file) to build portable and releasable binaries. You only need to type one command on a Docker-compatible host and another one on a macOS host to get engine binaries for Linux, Windows, macOS, and the cross-platform gamecode itself.

We build the Unvanquished release on Debian Buster and the produced Linux build must work on distributions from the Ubuntu Bionic era (2018, we made specific efforts for this), and the engine is still buildable on Ubuntu Xenial (2016): our continuous integration system verifies it for us and we tested such binaries were playable.

So, as you see, we make special effort to keep a very wide compatibility with our engine. Anyway, to not distribute obsolete binaries to the average user, if you really need to play Unvanquished on a five years old distribution, you’ll have to build the engine yourself. For that, see the Dæmon engine repository.

Fixing and improving the renderer

Between 0.51 and 0.52, an impressive list of bugs and design issues were fixed.

One impressive “small change, large effect” fix was a single rounding error that was the cause of many bugs. The related line was introduced in XreaL in 2005 and probably started to spit bugs in 2008 when other part of the code was modified…

This is not the only fixed bug that comes from middle ages… Another patch fixed a bug introduced in XreaL in 2006, and this time it’s sure, the bug was not dormant at the time. This bug was even older than Tremulous 1.1 (but Tremulous was not affected as it was based on ioquake3 instead of XreaL)… This bug was fixed 14 years later, day to day.

The Dæmon engine implements a GLSL shader cache since a very long time to save loading time, but for 0.52 we fixed bugs that prevented the cache to be properly invalidated in some situations. This fixed a range of “only happened once” mysterious bugs.

Unvanquished 0.52 introduces graphics presets and one feature used by those presets is that the engine now makes it possible to target a specific texture maximum size, instead of relying on the historic PicMip feature. PicMip feature was problematic because if you had a 64×64 pixels textures alongside a 4096×4096 one, the r_picmip 1 option would transform them to 32×32 and 2048×2048, one becoming uselessly super small and the other one still being huge. By setting the r_imageMaxDimension option to 512, the 64×64 texture is not modified, while the 4096×4096 one is turned into a 512×512 texture… Artists can use a special imageMinDimension material keyword to prevent the engine to downscale under a given size a texture known to suffer to downscaling. This is useful when used on textures with writing on them, or grates…

A huge revamp of the renderer code was done, more than 700 lines of code were deleted… and the engine does more things at the end! All the similar features now use the same code and not slightly different variants evolving from decades old copy-pastes.

The material file format now makes very easy to tweak some materials parameters like the normal map orientation: just add normalFormat X Y Z line and tell the renderer the normal map uses the OpenGL convention, or add normalFormat X -Y Z line and tell the renderer is using the DirectX convention. For historical reasons, without explicit keywords, the engine assumes the DirectX normal map convention when reading Quake 3 materials because XreaL (the engine we used as a base at the time) aimed to load unmodified Doom 3 materials and Doom 3 used the DirectX convention at the time even if it was an OpenGL game… All the open id Tech 3 derivatives, at least those that survived, followed this path.

We fixed many bugs, some being there from before Tremulous 1.1 exists… We also fixed some very old design mistakes we inherited from XreaL. XreaL was an impressive technical demo, but a technical nonetheless; it needed decades of work to be ready for production.

The historic forward renderer is now deprecated in favor of our tiled renderer, which is more performant when there is many dynamic lights in a scene, and this renderer is now in better shape than our previous one, the new one receiving all the care.

Starting with Dæmon 0.52, it’s now possible to run games in borderless windows, along the usual full screen and classic windowed mode.

Jumping into the future

Dæmon engine recommended texture packing guidelines.

Starting with 0.52, the Dæmon engine supports Physically Based Rendering (PBR). In a material file, just use the physicalMap keyword followed by the related image path to enable PBR for this material. The expected packing is ORM (Occlusion, Roughness, Metalness) similarly to the glTF 2.0 standard to make things easy. There was already a proof of concept in the 0.51 version but it was not made available. Currently the Unvanquished game does not provide PBR textures but this feature is believed to be ready to be used, including for community content.

We also fixed the relief mapping support. Relief mapping code was in engine since a very long time as it was inherited from XreaL, but it was unused and there was serious problems related to it. Some were even design flaws.

For some reason, the relief mapping code we had expected the height map to be shipped in normal map alpha channel upside down, this means white was down and black was floor level… After some investigation, it was discovered XreaL supported two ways to bring height maps: as loose files using the standard orientation, and as alpha channel in normal map files, but reverted.

The very big design issue about height maps being reverted while being in an alpha channel is that a non-existent alpha channel is meant to be equivalent to an opaque one, and an opaque alpha channel is expected to be fully white. So, with such design, a normal map file without alpha channel would make the engine consider the missing alpha channel is white, and because the height map was read upside down, a missing alpha channel would mean… maximum displacement. Yes, the way XreaL was designed meant that not shipping an height map in normal map alpha channel would mean maximum displacement.

To workaround that issue, XreaL implemented a special parallax keyword to use in materials, the lack of it would tell the engine to not attempt to read the alpha channel from normal map file. This was overcomplicated: to workaround a design issue with height maps being upside down, a special keyword was created to not be used so the engine knows it has to read normal maps with a code different than other textures.

By switching to the standard height map orientation (black is down, white is floor level), all the planets align properly: all normal maps files can be read as RGBA and height maps will always be fine even if the alpha channel is missing because no alpha channel means white, and white means no displacement… And then, the parallax keyword can be questioned.

Though, we made a special material keyword normalHeightMap we recommend to use with normal maps shipping height map, just to make it explicit and make it rock solid. One thing is that it’s very common in gaming to use a special format named BC3/DXT5 (or DXT5nm) which stores the X data in the Alpha channel instead of the Green channel. The engine properly handle them and does not consider such alpha channel to be alpha and then, everything works. But just to make it super sure, like preventing GPU driver bugs or others things, we recommend to use the normalHeightMap keyword when shipping both normal map and height map in the same file.

So, here are the two ways to add normal maps and height maps to a material:

		normalMap textures/shared_pk02_src/wall_big02a_n
		heightMap textures/shared_pk02_src/wall_big02a_h

or:

		normalHeightMap textures/shared_pk02_src/wall_big02a_nh

While it is supported to embed an height map in the normal map’s alpha channel, we recommend to store the normal map and the height map in two different files. This is because we recommend to use the DXT5nm format to compress normal maps and because this format stores normal data in alpha channel for efficiency, this format can’t ship height maps. At the same time, the Crunch tool will just pick the best DXT variant for loose grayscale images as well. When disabling relief mapping (which is costly), the height maps would never be loaded from disk neither uploaded to GPU memory.

The engine now supports height map scale and offset, though this is not yet officially available to artists. The reason is that we used the Xonotic maps and textures as a testbed for those features, so this is currently only supported in DarkPlaces compatibility mode which is disabled by default, and using keywords that cannot be used on blended terrains for example. The DarkPlaces keyword issue is similar to the one from Doom 3 we talk about below, stages are not fully supported and then keywords may not be designed to be compatible with such feature. Providing support for heightmap scale and offset is planned anyway. Most of the code is already there and working, we just need to add the related material keywords so artists can use them.

As we said, we added a material keyword to switch normal map component orientation. We haven’t investigated why XreaL expected height map embedded in normal maps to be reverted. We know that XreaL reverted Y normal component because of mimicking Doom 3, maybe XreaL reverted height maps because of a game, and was just reproducing a design issue from an implementation reference… If needed it will be trivial to add a keyword to switch the height map orientation anyway.

We created a special repository of assets purposed to test feature support and prevent regressions in the engine and related tooling (NetRadiant level editor, Q3map2 map compiler…). For example there is maps (levels) for testing the various normal map formats, to test for dynamic lighting, for all the various PNG image format variants, etc. I want to specifically thank SomaZ from the OpenJK project for the assets he provided to test PBR support.

We implemented code to compute normal map from height map if the normal map is missing. The code works and is probably ready to ship but it missed the 0.52 release because it lacked a proper test map to make sure the computed normal map orientation is the expected one. You’ll see that code soon anyway. That said, we would not recommend to purposely rely on it as this would be heavy on the GPU, but this can improve compatibility with third party content.

Because we used Xonotic assets as a test bed for some new features we implemented or fixed, the Dæmon engine supports basic symbolic links in PK3 / DPK archives, and provides a DarkPlaces compatibility layer, though sRGB light maps are not implemented yet. Previously in 0.51 we also implemented optional dds/ prefix for DDS files like DarkPlaces and Doom 3 expects them.

For anything that must not be compressed with DXT formats, WebP is perfect.

Talking about DXT formats, the recommended image formats to use with the Dæmon engine are those ones:

• Light maps: Lossless WebP;
• Skyboxes: Lossy WebP;
• Normal maps: Normalized CRN (Unity/Dæmon variant with -rtopmip option);
• Everything else: CRN (Unity/Dæmon variant).

The CRN format is a special format optimized for compression and visual quality. It is produced by the Crunch tool. Initially developed by Binomial, it was unmaintained and lately Unity greatly improved it. As told at Unvanquished 0.51 release time, our whole corpus of 1797 textures at the time was compressed 4.31 times faster and 11.15% of the produced size was saved, as promised by Unity guys. The Unity variant is incompatible with the Binomial one but given the numbers, there was no reason to not make the jump. Because both BinomialLLC and Unity are just dropping code without expecting community participation, and not taking that much care about multi-platform support, we maintain Crunch on our GitHub. Outside of the quoted -rtopmip option for better normalization of normal maps, all we do is to maintain cross platform support while not breaking compatibility with upstream. If you look for a maintained Crunch you can integrate on Windows, Linux and macOS and/or look for proper CMake integration, you just found it.

Fixing material design issues

We fixed a texture rotation bug. This was first noticed with some Tremulous maps like Metro or our own Unvanquished map Parpax. We then reproduced the bug with Smokin’ Guns assets, and gimhael fixed it.

We also fixed an issue with special skyboxes with faces not having the same size. Yet again this bug was reproduced with Tremulous maps and Smokin’ Guns ones. Having skyboxes with faces not having the same size may be surprising, but it looks like it was an optimization trick used at Quake 3 time: the bottom face is usually under the floor and never seen (unless your map is a space floater), so the bottom face was sometime a very small placeholder. Given fixing this issue uncovered many potential others, in the end we entirely rewrote that part of the skybox code.

Speaking about materials, we modified a bit the syntax to fix a design regression coming from Doom 3. When XreaL implemented multitexturing like adding normal maps, specular maps, etc. to a single texture, it implemented the Doom 3 syntax.

At Quake 3 time it was possible to describe a simple material like a list of stages. For example one stage paints the wall texture and another stage paints the computed lights and shadows. Those stages were powerful: there are keywords for blending, rotating, translating. They were not called materials but shaders… Adding an addition map would just need to add another stage to blend it over the previous ones.

Though, for most of the uses cases, like a simple surface with a single texture expected to receive light and shadows, this was very verbose, you literally had to write blend operations for that…

Doom 3 created special keywords that were shortcuts for stages, a simple line like diffuseMap path/to/diffusemap was configuring the diffuse map being applied with computed lights, and specularMap path/to/specularmap line was calling the specular stage.

Because rendering multiple passes is costly and one GLSL shader may run multiple operations at once, engines like XreaL then Dæmon attempted to collapse materials with some heuristics, so a single GLSL shader may compute the normal, blend the lights and the addition map, etc. It means the syntax was suboptimal and did not translated well to the real computation, even if it was possible to do some guessing at material parsing time to mitigate this.

But this syntax was not only suboptimal because it did not fit the real computation, it was also suboptimal because it did not fit the user’s needs, and one issue that was not fixable with optimization code was feature regressions.

Quake 3 supported a feature that is called terrain blending. You can stage multiple textures in a single material, like one rock and one sand textures, and with some computation, the map compiler and the engine will blend the textures on the surface. This is really useful for terrains.

But with Doom 3 syntax, a specular stage would live at the same level of a one or the other blended texture, so the specular stage for a rock texture would be applied on a sand texture as well if the material is about blending rock and sand.

Multitextured terrain blending on Hangar 28 community map by tvezet, with diffuse, specular, normal and height components.

So, before Dæmon engine 0.52, a game was able to provide multitextured surfaces except in some cases like terrain blended ones that were still rendered like in 1999. Such a shame…

We improved the syntax by making keywords like diffuseMap not describing stages, but describing components of a single texture used in a stage.

So the Dæmon engine supports the legacy Quake 3 syntax (that can still be useful for the 5% of specific uses cases), the Doom 3 syntax (with warnings because there is no good reason to use it), and the Dæmon syntax, with two examples given there:

textures/parpax_custom/squarelamp_blue_40k
{
    // some hidden editor and map compiler keywords
    {
        diffuseMap  textures/parpax_custom_src/squarelamp_blue_d
        glowMap     textures/parpax_custom_src/squarelamp_blue_a
    }
}

or:

textures/thunder_custom/ter_rocksand_xy
{
    // some hidden editor and map compiler keywords
    {
        diffuseMap  textures/shared_pk02_src/rock01_d
        normalMap   textures/shared_pk02_src/rock01_n
        specularMap textures/shared_pk02_src/rock01_s
        rgbGen identity
    }
    {
        diffuseMap  textures/shared_pk02_src/sand01_d
        normalMap   textures/shared_pk02_src/sand01_n
        specularMap textures/shared_pk02_src/sand01_s
        blendFunc blend
        alphaGen vertex
    }
}

Starting with the 0.52 version, the Dæmon engine offers the powerful versatility of Quake 3 shaders, while at the same time it makes easy to set-up multitextured materials like Doom 3 materials, while avoiding performance shortcomings related to Quake 3 shaders (no extra renderer passes, no collapse heuristics to save renderer passes…) and avoiding limitations of Doom 3 materials: you can blend multiple textures having more than one component! On the syntax side, it’s only a matter of extra {} blocks. On engine side, the difference is like night and day.

The collapse code for legacy materials was entirely rewritten from scratch. Huge parts of the material parser were rewritten, same with parts of the code configuring the GLSL shaders, and some large parts of the GLSL shaders themselves were rewritten as well.

A view on the Dæmon engine ecosystem

The Dæmon engine currently uses Native Client as a platform to run our and your own game code with native performance. Build your gamecode once, ship it to the players or host your mod on your own server, and players on Linux, Windows and macOS will enjoy the game.

Unlike some older technologies like the original QVM which required a now-obsolete non-free specific C compiler, you can write modern C++ code and rely on well-known on-the-shelf C and C++ libraries for your game, while still ensuring players run your custom code sandboxed. And you don’t have to mind about the operating system your players use.

Because the industry is moving from Native Client to WebAssembly, we are currently in the process of migrating our engine as well. So other architectures than i686 and amd64 are currently on hold because we don’t invest time in Native Client anymore and we focus on moving to WebAssembly first before investigating other platforms, so as to save precious development time. For people wondering about Apple Silicon support, before we complete WebAssembly and ARM support, we can reassure you: the Dæmon engine and the Unvanquished game work properly on Rosetta 2 compatibility layer.

Unlike things like the Godot engine or the Unreal engine, the Dæmon engine is not an editing platform by itself, it is the raw engine to run the game, a component that fits in a wider ecosystem of free and open source tools around the engine to fulfill your needs, including the NetRadiant level editor and others.

Alongside the legacy Wav, TGA, PNG and JPEG audio and image formats, the Dæmon engine supports Vorbis and Opus audio compressed formats, and supports WebP image format and GPU-optimized image formats like KTX, DDS and CRN (see our maintained version of Crunch). CRN is perfect for almost every textures of the game, while we recommend lossy WebP format for textures that may suffer from aggressive lossy compression like skyboxes, and lossless WebP for textures that must better not be lossy, like light maps.

We dropped video support for now because it was implementing a format that does not exist and must not exist: OGM with Theora. OGM was a hack to mux XviD video with Vorbis decades ago before OGG was standardized for Theora and Vorbis, so neither OGG tools neither OGM tools can produce OGM files with Theora. This code was living in an alternate reality that never happened. If someone needs video support again, it would be better to implement WebM with VPX and Opus, given our engine already implements WebP and Opus.

Tools like upstream NetRadiant and the related Q3map2 map compiler and lightmapper support those advanced texture formats as well, so you know you can deliver optimized assets to players. Players can start mapping community maps for your game just after having downloaded the game itself and the editor, nothing more.

Alongside the legacy Quake 3 md3 and Doom 3 md5 model and animation format, the Dæmon engine supports the Inter Quake Model format, which is the de facto standard for open source game engines like id Tech derivatives or Cube/Sauerbraten ones. IQM is open source, documented, support features like bone-based animations, vertex colors, etc. and there exist various viewers, importers and exporters for this format.

With the NetRadiant level editor, the Q3map2 map compiler, and the Crunch texture compression tool, we have a graphical tool named Chameleon to assist in the process of retexturing levels with higher definition textures, the Sloth tool to automatically write Dæmon materials for texture repositories following related naming conventions, and Urcheon to handle the process of building a data source repository and producing a working DPK archive: you just type urcheon build src/map-castle_src.dpkdir and you get a usable dpkdir with maps being compiled to bsp, textures compressed to this or that variant of WebP or CRN given what they are for, models translated, rotated, etc. and converted to IQM, audio compressed as Opus… This is like cmake for data packages, but without the need of writing explicit CMakeLists.txt, instead it relies on naming conventions to pick the best action for your data. For IQM models we rely on the IQM branch hosted by FTEQW because it has support for command files and working translation/rotation/offset features.

The engine implements an enhanced virtual file system named DPK VFS that is similar to the PK3/PK4 VFS, but with the ability to only load a given set of packages. This way a mod or a custom map can rely on given resources, packages, or texture sets, without worrying about what the other mods or maps do.

How to reduce hidden costs

If you plan to make a game with the Dæmon engine, and this is true for any engine of this kind (ioquake3, DarkPlaces, RBDOOM3-BFG…), keep in mind that despite it looks super easy to clone and to build its own fork, and of course it is, maintaining a game engine is another task and it quickly turns into a full time job.

And then you would still have to find time for the game development itself, the gameplay design, the level editing, the modeling, the sound design, the texturing, the communication of your project and other public relations, the release cycles, the bug tracking and fixing, the testing, the publishing…

So if you want to take the least amount of risk and want to see your game becoming successful we deeply encourage you to contribute your own patches for your own needs to the upstream Dæmon project.

Git, CMake and usual tools make it so easy to build things they may give false impressions about many hidden costs. This is a game engine tailored for production, even John Carmack was working full time when he worked on id Tech 3 at the time, think about it twice. 🙂

For the days to come

So, for the future, there is the ongoing WebAssembly effort, with slipher and Kangz working on it, Kangz was the one who already did the port from QVM to Native Client, so we know he has the skills to do so, all he needs is some free time.

There was experiments done to support better colorspace for light map storage to reduce color banding, as well as dithering for the same purpose, there is also non-merged yet code to generate normal maps from height maps when normal maps are missing, and some fallbacks to enable deluxe mapping on materials without normal and specular maps are investigated. Time will tell, but it’s probable some of those features will make it to a next release.

And the best thing that is coming is that next Friday, on May the 14th, we will release Unvanquished beta 0.52, based on this engine. Tell your friends! Bring yourself! Get ready!