Missing libraries Thread - Desura

Some games don't run on specific OS due to missing libraries.
Nothing happens when starting such game, only error message prints to terminal.
Running desura from terminal is not normal, I think. It is possible to implement hooking of such messages and display error message in a window?
It would be nice if error message window would contain short help and web links.

The problem is that some developers have their games exiting with an error code. So capturing errors for every game with end up with some who always show an error window.

If you let me know what Linux distros don't have specific libraries for specific games then I can add them to the game on the server.

Windows always show such error message if dll was not found. Why not to make it for Linux?

Like I said some games in Linux exit with error codes. Also the Windows version doesn't just show DLL errors but all errors. So that is the problem when games exit with errors.

I haven't seen any working application with missing shared library.
If game runs but fails to load libraries at runtime - it's another situation that should handled by game itself.
We are talking only about missing libraries that game requires, not about errors or something else.

Edited by: Marzanna

I'm using Ubuntu 11.10 and I got some missing libraries. I only found it out when I tried to run the games from the terminal, which printed the following messages:
./opentyrian: error while loading shared libraries: libSDL_net-1.2.so.0: cannot open shared object file: No such file or directory
./smokinguns.x86: error while loading shared libraries: libopenal.so.0: cannot open shared object file: No such file or directory
./CoreBreach: error while loading shared libraries: libgnustep-gui.so.0.21: cannot open shared object file: No such file or directory

OpenTyrian had it checking for and installing llibSDL_net. So you might need to uninstall it and install it again.
Smokin' Guns also tests for and installs libopenal if your system doesn't have it.
CoreBreach also includes the library you are talking about in the game's folders when you download.

It sounds like there is something very weird going on with your setup that it isn't picking up these libraries like it does for everyone else. I use Ubuntu 11.10 AMD64 and have none of these issues that you mention.

SkaarjZR wrote: Some games don't run on specific OS due to missing libraries.
Nothing happens when starting such game, only error message prints to terminal.
Running desura from terminal is not normal, I think. It is possible to implement hooking of such messages and display error message in a window?
It would be nice if error message window would contain short help and web links.

+1 for the idea! It could be just something like:

'An error occured. For details click below {down a button to unwrap log from terminal}. Please consider also checking game logs.'

For some reason I can not run any games (Kubuntu 11.10 AMD64) except for the Core Breach demo, presumeably because those needed libraries are included. Today I tried to install OpenTTD and Towns. They get just to the end of the download, and then error out. So, Protektor, I'm assuming that there are differences between a stock Ubuntu install and a Kubuntu one. But I'm not sure how I can find out for sure.

I did check and libSDL_net was not installed on my system. Installing now and will test again.

Update: Reinstalling Desura after deleting all folders and running from the CLI (./desura) gives the following console warning:

(desura:25126): Gtk-WARNING **: /home/ronnoc/Games/desura/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib/x86_64-linux-gnu/gtk-2.0/2.10.0/engines/liboxygen-gtk.so)

I have libstdc++6.x installed, so I'm at a loss here.

Oh...someone else had a problem with Oxygen. If you do a search of the forums you should fine the answer if that is indeed the problem your having. It also sounds like you are missing some of the c++ support libraries as well for some reason.

Do you do a lot of install and uninstalling of software?

It seems to me like liboxygen was compiled with a newer C++ compiler than the game. I sounds like the basic problem where you have to bundle all C++ code to remain portable. If a program pulls in any library from the system that was compiled with an incompatible g++ version, you will see problems.

It seems to me that the issue is that I'm running KDE, which uses a lot of Qt and no GTK+. What else could it be? Kubuntu is identical to Ubuntu otherwise. I'd like to use and reccomend Desura, but since I can install any of these games and run them outside of Desura, the problems I'm having are with the client. For example I have a thread here about Smokin' Guns. This game installs and runs just fine outside of Desura. Same with Towns, Urban Terror, etc. etc. None of them run insid Desura. They download to 100 pct and then error out.

Could be, but the error about "GLIBCXX_3.4.15" say a needed library exists. It was just compiled with the wrong version of the compiler.

I uninstalled and installed the games again and it did not fix the problem.

What distro and version are you using?

Sorry! =)

Desura build 110.22 under fully updated Ubuntu 11.10 32-bit.

Protektor wrote: The problem is that some developers have their games exiting with an error code. So capturing errors for every game with end up with some who always show an error window.

The linker will always exit with status code 127 if it fails to load a library. It's possible for a game to exit with the same code if it really wants to, but that's probably rare.

Protektor wrote: OpenTyrian had it checking for and installing llibSDL_net. So you might need to uninstall it and install it again.
Smokin' Guns also tests for and installs libopenal if your system doesn't have it.
CoreBreach also includes the library you are talking about in the game's folders when you download.

It sounds like there is something very weird going on with your setup that it isn't picking up these libraries like it does for everyone else. I use Ubuntu 11.10 AMD64 and have none of these issues that you mention.

It'd be much better if games always installed libraries they depend on. Even leaving aside the whole different versions and interoperability argument that I've talked about before, if libraries are only installed by Desura if they're missing on the system at the time they're installed, you get into situations where someone has a distro package installed that depends on e.g. sdl-net, then installs a Desura game that also wants sdl-net, and everything works. Then this person uninstalls the package that depends on sdl-net, and sdl-net is also uninstalled since there's nothing else that depends on it as far as the package manager is concerned. The Desura game is now broken!

Solving this problem is not hard! Just include all libraries that a game depends on with the game, and point the linker to them with either RPATH or LD_LIBRARY_PATH. You don't even have to bug the game developers about this if you don't want to (though you probably should so they can fix it upstream). It's easy to detect missing libraries and add them where they're needed. Yes, this means that a game installation might be a few megabytes extra compared to before, but it'll work without having to mess around with dependencies, which is a problem something like Desura's supposed to solve for us. (Not to mention that even the smallest hard drives today are measured in hundreds of gigabytes, a few extra megabytes for this extra convenience is a tiny price to pay.)

Bah, sorry about the rant. I really want to like Desura, but it seems like such wasted potential at the moment. Problems like these are simply unnecessary!

Edited by: mjau

mjau wrote: Solving this problem is not hard! Just include all libraries that a game depends on with the game, and point the linker to them with either RPATH or LD_LIBRARY_PATH.

I agree. Just bundle the libraries. I don't understand why this trivial(?) problem bites us all the time.
IME, if all libraries are bundled, Linux games work for longer time without problems than Windows games.
For example, I can simply run and play ancient Loki games with gentoo "unstable" (kernel 3.2, gcc 4.6, glibc 2.14, mesa 8.0 opengl) without any problems. Sound works, Opengl graphics work, input works. And the builds are now over 10 years old!

dequire wrote: Update: Reinstalling Desura after deleting all folders and running from the CLI (./desura) gives the following console warning:

(desura:25126): Gtk-WARNING **: /home/ronnoc/Games/desura/lib/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib/x86_64-linux-gnu/gtk-2.0/2.10.0/engines/liboxygen-gtk.so)

I have libstdc++6.x installed, so I'm at a loss here.

The problem is that the libstdc++6.so.6 which comes with desura is too old for your system. The liboxygen-gtk.so is your system library and it needs GLIBCXX_3.4.15 while desura-bundled libstdc++6 provides only GLIBCXX_3.4.14). As a workaround, you could try to copy your system libstdc++.so.6 to desura/lib/ .

Edited by: pttr

The trick is, it's not always safe to bundle every libraries. There are even some you never want to bundle. I would vote to always include the libstdc++6 (and libgcc_s) the game was compiled against along with and other libraries that have C++ code in them. But Desura and game developers have been bitten before by bundling something that caused trouble on certain distributions.

Simply put, it's not fair to make it seem like Desura is just being difficult. They've been working through this some what by trial and error. There is no simple guidelines as to what must be bundled, and what must not be bundled. (If anyone can declare a clean set of rules, I'll modify a tool I started to scan a program for compliance.)

Edited by: Eddward

I didn't mean to say that Desura is being difficult, thats why I had the question mark in there.
That tool seems to have good set of rules. Most if not all library problems seem to come from just missing libraries or libstdc++6 and library incompatibility.

Eddward wrote: It sounds like the basic problem where you have to bundle all C++ code to remain portable. If a program pulls in any library from the system that was compiled with an incompatible g++ version, you will see problems.

Oops, after reading this thread again I can see that the problem and the solution are already clearly pointed out.

Eddward wrote: Simply put, it's not fair to make it seem like Desura is just being difficult. They've been working through this some what by trial and error. There is no simple guidelines as to what must be bundled, and what must not be bundled. (If anyone can declare a clean set of rules, I'll modify a tool I started to scan a program for compliance.)

I didn't mean to imply that Desura is being difficult on purpose, but the problem really isn't as hard as most people seem to think. I admit I simplified it a little bit before though. Anyway, here's what I do:

• Don't bundle glibc (libc, libm, librt, libdl, libpthread, etc), the linker (ld-*.so), or system specific libraries/drivers like libGL, libasound, etc. (Stuff like X11 and GTK are more of a grey area; probably better to not depend on those and use dlopen in stead)
• Statically link what you can. Specially libstdc++ and libgcc is good to link statically if you can (license has an exception that allows this even for closed source). You can't statically link those if you need to pass exceptions between several libraries that can't also be statically linked though.
• Bundle everything else. Point linker to libraries via either RPATH or LD_LIBRARY_PATH. Try not to mix and match, bundling some and relying on system libraries for others; really do bundle everything else if you can.

I don't claim to be an authority on this or anything, but I've more or less been doing this with DROD since 2003, and while there have been some issues with libraries needing an update from time to time (e.g. when SDL added pulseaudio support), it's worked perfectly otherwise.

Also, SDL is pretty nice: When configuring it for compilation, you can select to dynamically detect and load at runtime any system-specific libraries it might need (OpenGL, ALSA, pulseaudio, X11, etc), using dlopen. This is perfect, because libSDL won't require any of those to be present to work, but it'll still support them and can use them if they're available.

That's about it, except the issue with glibc versioning. Usually this won't be a problem, but sometimes you may get a build that depends on a "too recent" version of glibc, like e.g. depending on glibc 2.12 is probably a bit too restrictive at the moment. This can't be solved without a rebuild though. Also, gcc unfortunately doesn't have any switches that let you decide which glibc version to depend on (or didn't last time I checked, anyways), so you have to either compile against an older glibc, or override the symbol versions using a pre-included header when compiling. You'd have to do this for both the exectuable and any libraries that depend on too recent glibcs. (I've got a script that can generate the pre-include for you: gensymoverride (a bit rough, but does the job..). Include the file it generates when compiling by adding "-include myfile.h" to CFLAGS or something. Maybe also add "#define _FORTIFY_SOURCE 0" to the top if you get complaints about missing __*_chk symbols. At the moment I use this to not depend on anything later than glibc 2.7 (shipped in the previous stable version of debian), but you can probably go a bit more recent than that without too many problems).

Eddward wrote: (If anyone can declare a clean set of rules, I'll modify a tool I started to scan a program for compliance.)

I checked this out and noticed you're using ldd to find dependencies. You'd probably have more luck using objdump for this (objdump -x elf | grep NEEDED for required libs), since that will only list immediate dependencies, while ldd gives you the whole chain, including things required by libraries that you won't include in the first place. You'd have to run objdump on all the libraries you want to include too, though. Actually, I recently made a little tool that does just that and then copies all dependencies of an executable to a particular dir: depcopy (requires lua). Note that the skiplibs table in that is incomplete, that's just what I needed for the most recent DROD while also sourcing from a dlopen-enabled SDL. Anyway, feel free to grab ideas from that if you want =)

Edited by: mjau

For what it's worth, I only use ldd to build a mapping of library names to file names. I didn't want to search the system libraries paths myself. I use 'readelf -d' and grep for NEEDED to get per-object dependencies. I can switch to objdump if it's better.

I think one of the bigger issues is when people link against system version of the libraries at compile time. I forgot about that. I think LGP has said they build custom versions of libraries. I know Troy Hepfner of "My Game Company" has posted a series of articles for supporting Linux. He also covers compiling custom version of libraries to cut down unneeded dependencies in the second article "Distributable Binaries". This is probably a good cookbook to follow.

In any case, it's an issue for each game developer and probably not Desura. I wonder how hard it would be to make a simplified system for games devs that has 'recipes' for build common gaming libraries. Something like gentoo ebuilds but with a more complete build/packaging solution. Something like Yocto is supposed to be for the embedded world. It could be something where the dev defines what they want. The tool could then fetch and build a toolchain and distributable libraries. The developers could then just build their game against that toolchain and have it spit out a mojo file and maybe an rpm & deb file. Sound crazy?

Eddward wrote: For what it's worth, I only use ldd to build a mapping of library names to file names. I didn't want to search the system libraries paths myself. I use 'readelf -d' and grep for NEEDED to get per-object dependencies. I can switch to objdump if it's better.

Oh, I'm sorry! No, readelf is fine. (The objdump output is a bit easier to parse, but it's the same information). There do seem to be a lot of extraneous libraries listed though, but I guess that just comes from using system libs in stead of specially built ones.

I think one of the bigger issues is when people link against system version of the libraries at compile time. I forgot about that. I think LGP has said they build custom versions of libraries.

Oh, right, I forgot to mention that too. [e: actually though, what you build against shouldn't matter as long as the ABI stays the same; which libraries you ship with the game is more important.]

In any case, it's an issue for each game developer and probably not Desura.

I agree, but the last time someone brought this up, Desura replied with something along the lines of "it's already difficult enough to get developers to release their games on Desura, and we don't want to impose more requirements on them to make it even more difficult". Personally I think that's a bit short sighted of them, but anyway, that's why I suggested ways Desura could at least partially fix this without having to involve developers if they didn't want to.

I wonder how hard it would be to make a simplified system for games devs that has 'recipes' for build common gaming libraries. Something like gentoo ebuilds but with a more complete build/packaging solution. Something like Yocto is supposed to be for the embedded world. It could be something where the dev defines what they want. The tool could then fetch and build a toolchain and distributable libraries. The developers could then just build their game against that toolchain and have it spit out a mojo file and maybe an rpm & deb file. Sound crazy?

No, that doesn't sound crazy at all! (Haven't heard about Yocto before, will have to check that out). I think this would definitely be a good thing.

Currently I'm using gentoo in a chroot (actually 2, x86 and x86-64), and while that's not strictly necessary to build compatible executables it does help a lot in the long run. It's a bit complex and time consuming to set up right though, and even after you've done that there's a lot of small annoyances that pop up that really shouldn't be a problem at all. I've had a few problems with udev, for example, which is just silly since I have no use for udev in the chroot, it's just a gentoo system dependency. I also had to make my own ebuild for cmake because it insisted on depending on an ssl-enabled curl, which I built without ssl to cut down on dependencies (not to mention that I don't need cmake to be able to download anything in the first place). The SDL ebuild had to be modified to enable use of dlopen, since there's no USE flag for that. Etc, etc. Small things, but they can add up.

So anyway, yeah, something like what you propose would be great. Also, if it would allow you to build and manage several different versions of a library at once and select between several "profiles" (collections of libraries) at build time, that would be fantastic. That way you could easily have different build options of the same library for two different projects. Would need to ensure that the compiler will always pick up the right includes and such though. A chroot gives you that automatically, but it'd be great if this would work without resorting to that. (Could maybe fake a "soft" chroot by intercepting glibc/kernel calls, but it's probably doable without that, too).

Edited by: mjau

I've started experimenting with yocto. A fellow on their IRC what their meta-toolchain stuff might work. I'm not sure.

It might be easier to abuse portage and make custom ebuilds without the un-need dependencies. I'd like to avoid the need to chroot and just have it build a toolchain that installed and operated out of a given sub-tree.

We probably ought to move this off this thread, but I'd prefer not to drop it. If there's a way to make it easier to build portable game binaries on Linux I'd like to explore it some more.

Take a look at Gentoo Prefix -project for tips and scripts for abusing portage.
It looks like good starting point for building custom toolchain to some subdir.

GentooPrefix wrote: The Gentoo Prefix project develops and maintains a way of installing Gentoo systems in a non-standard location, designated by a "prefix".

Gentoo.org

Edited by: pttr