Downloading data files from the server

To kick off the multiplayer improvements for version 2.1, I’ve started with adding access to data files over the network. For example, if a server is running a custom PWAD that you don’t have, the client will automatically download a copy before joining the game.

That pretty much sums it up for the user-facing portion of this feature. You will see a popup displaying download progress, and there is an option to cancel.

Much of this work was recently merged to the master branch (in the form of 83 commits) and is included now in unstable builds. Note that this kind of a larger influx of changes usually leads to new glitches also being introduced… I will be improving the code in the coming days/weeks.

Continue reading Downloading data files from the server

Doomsday 2.0.3, other minor updates

Over the past few weeks, I’ve made a couple of stable builds with the version 2.0.3. The first one of those was made a bit early as I was about to leave for a trip abroad, and mainly included one bug fix for Hexen related to saving and restoring object state from save files. Recently I’ve made a couple of additional stable builds to investigate and fix a problem with the Ubuntu Launchpad build scripts, where the “doomsday-stable” packages were correctly built but nothing was actually included in the generated DEB packages.

On the whole, progress has been somewhat slow. Perhaps the biggest advance was in the dengine.net website backend, where I’ve now split the API functionality to a separate api.dengine.net server, so that things like master server and update queries won’t interfere with the normal operation of the project home page and forums. I hope this will alleviate the issue of dengine.net sometimes failing to respond to requests.

Prompted by a forum post, over the weekend I was investigating an audio volume issue on Windows. It turns out there is a problem with the SDL_mixer music volume controls. I have yet to determine if there is a workaround that Doomsday can do to avoid the issue. Such a workaround would be preferable to disabling SDL_mixer music on Windows completely, since SDL_mixer does bring value to the table (e.g., music formats). The situation is also slightly tricky because SDL and SDL_mixer are built in to the engine, so there isn’t a plugin to take out or something simple like that.

Odds and ends

I’ve been getting back into the swing of things with a set of smaller changes and improvements. I haven’t yet started work on the major focus of the 2.1 release, which is planned to be Multiplayer improvements.

2.0.2 will be out on Saturday, July 1st, with a couple of bugfixes that I hope will be useful. As usual, I will post the details after the build is available.

The recent changes include:

  • Stability improvements. Several methods that went against the C++ standard were revised. Reliance on undefined behavior was leading to compiler optimization issues particularly with the latest GCC 7. Also, error reporting is now more robust. For instance, previously when something went wrong during busy mode (like engine startup), it could easily lead to a crash because the client app wasn’t reacting to the situation appropriately. When the error message was being shown, the application was still trying to do something in the background.
  • More powerful model scripting. Starting animations and timeline scripts via Doomsday Script.
  • The model renderer shaders support more macros for easier customizability. Generally speaking, using macros is more future-proof than writing completely customized shaders.
  • Resource identification fixes. There was a duplicate IWAD spec for Heretic 1.3 vs. SOSR. Now only the latter remains; if you have problems with your Heretic IWADs, Clear Cache and restart Doomsday. Also, autogenerated package IDs had a problem with special characters in file names. Doomsday will choose package IDs based on the file name of the data file, however it allowed spaces and quotes be included in the IDs. This lead to problems because whitespace characters are used as separators in lists of IDs, and quotes may not be accurately escaped in Info strings. Again, Clear Cache is recommended.
  • Assimp build option. There’s a new CMake build option (DENG_ASSIMP_EMBEDDED) for disabling the use of Doomsday’s customized version of Assimp. This allows one to configure Doomsday to use the Assimp installed as a system library, which is more common on Linux for instance.
  • More UI tweaks. Plenty of small things:
    • Minor dialog layout improvements to avoid awkward dimensions.
    • Changed the UI font on Windows.
    • Added popup outlines for visual clarity.
    • Fixed Home tab scrolling with the mouse wheel so that it doesn’t always jump to the previously selected item when switching between tabs.
    • Game icons are refreshed when package availability changes. No more missing and incorrect icons shown for game profiles.
    • Fixed missing window fullscreen state notification (was affecting the appearance of the “Quit” button on Mac).
  • Monitor refresh rate. Yesterday I started working on monitor refresh rate configuration options. I realized these have been completely missing from the settings for some time now. Doomsday does query the supported display modes, which includes information about refresh rates, so now you have the option of choosing which refresh rate Doomsday will prefer when switching modes on Windows/Linux (display mode changes are not done at all on macOS). However, I recommend you consider adjusting the game pixel density instead of the screen resolution to keep the UI sharp and crispy.

Notes about build 2353

I’ve been taking it easy for the past couple of weeks, but last weekend I did manage to merge the recent OpenGL changes to the master branch. Build 2353 now includes this work. Please let me know if something appears broken!

There was a slight problem with the autobuilder, so the build change log is empty. GitHub has your back, though.

Here’s a rundown of the most important changes: Continue reading Notes about build 2353

Planning ahead for 2.1: multiplayer and OpenGL

The stable version 2.0 has been out for a bit over two weeks now. I hope you’ve been enjoying it! I have been busy working on bug fixes for 2.0.1, which is soon getting into shape for release.

Now that the goals for the 2.0 release have been met, it is time to look ahead to version 2.1. The first point release is planned to primarily feature improvements for multiplayer games. I won’t be exclusively working on that, though, so various minor improvements will also be included. Small things like a persistent console command history and options to start a game profile in a chosen map are already available in the unstable 2.1 builds.

When it comes to the release schedule, it remains futile to speculate about how much time I will have for Doomsday in the future, but my hope is that point releases in the 2.x series will have a roughly six month cadence. While there is no specific release date set for 2.1, I am uncomfortable with the idea of more than 12 months between stable releases. If that much time passes I am willing to postpone the remaining work to a later point release and do a smaller stable release instead. I prefer that there is a steady and predictable flow of stable releases, much like the situation is with the unstable builds.

In any case, if you run unstable builds you will of course get all the latest improvements as soon as they are implemented.

Multiplayer

2.1’s primary objective in terms of user-facing features is multiplayer improvements. This means both bugfixes and new features. I envision the following MP-related improvements in concrete terms (list still subject to change, in no particular order):

  • Bug fixes. Multiplayer gameplay is suffering from a variety of small glitches, particularly in Hexen. The goal here is to get rid of the more significant ones.
  • Movement smoothing for enemies. Single-player games use movement smoothing for everything, so MP games should be no different.
  • New chat UI. Modern games have some established conventions regarding in-game chat, so Doomsday should take inspiration from them. The chat should use Doomsday’s native UI widgets so the font isn’t blurry or too large, text can have colors and rich formatting, there is nice scrolling, text entry is more versatile, and there is autocompletion for usernames and commands. One particular direction this feature could take is integration with the console command prompt, so one can still issue console commands using IRC-like slash prefixes while primarily still chatting.
  • New scoreboard UI. A better scoreboard would be a nice thing to have for competitive game types. There should also be an option for a small, always-on scoreboard HUD. I will likely end up with a toggle between completely hidden, small HUD, and full-size scoreboard modes.
  • More GUI options for configuring game via Shell GUI. Most of the gameplay cvars should be configurable via the GUI. The 2.0 Shell already offers a couple of options (such as game type and current map), but there are more options that would be useful to have conveniently available. My strategy with regard to the Shell remains unchanged, though: for hosting games, the Shell should be the tool you use. Duplicating its features in the client GUI would be more or less redundant and wasted effort. However, I can look into integrating the Shell better with the client GUI, so it is quick and easy to get a local server started, for instance, by launching the Shell via the client.
  • GUI for configuring map cycle. Map cycling is one important MP feature, especially for longer-running servers. By default, a co-op server will not restart the episode once it ends; map cycling should for instance allow configuring entire episodes to be played through and then restarted.
  • Remote packages. The key feature here is loading PWADs from the server.

OpenGL

The internal objective for 2.1 is to complete or at least take a major step toward migrating the rest of the OpenGL code to version 3.3, making it also OpenGL ES 2 compatible. GLES is a requirement for Android and iOS, and also Raspberry Pi (a good low-powered testing/development platform). As a nice side-effect, this transition will also future-proof much of the code for subsequent migrations (particularly thinking about APIs like Vulkan, or some other low-level graphics API of the future). The crucial advantage that this transition will provide is support for GLSL shaders when rendering world surfaces. This truly opens up the potential for renderer improvements.

However, in the scope of 2.1, it must be noted that this will be an internal change, meaning the renderer features available to you remain roughly the same as in 2.0. Some older GPUs may no longer be supported, although it is also possible that some OpenGL drivers work better with OpenGL 3 instead of the old compatibility profile. It may also be necessary to drop some older graphics settings that are no longer relevant, if it isn’t feasible to reimplement them with new GL code.

Finally, please remember that plans may change so none of this is particularly set in stone. Stay tuned for updates. 🙂

Website refresh

In preparation of the 2.0 release of Doomsday I have redesigned the dengine.net website from the ground up. The new design went live today.

While the old design has been perfectly adequate during the past years, I felt that a bit of freshness would do the site good. I also took the opportunity to tie all the parts of the site together and reconsider the overall organization.

There is a new navigation bar visible on all pages, even in the Autobuilder and Bug Tracker that have previously been detached from the rest of the site. This should make it easier to move around and find what you’re looking for. In the bottom of most pages, you can find the latest news and blog posts, recent builds, and currently running multiplayer servers — these were previously only shown on the dengine.net front page.

Following the migration of the forums, the Doomsday Engine Wiki has been replaced with the more focused and structured Manual. It uses DokuWiki, which makes it considerably less complex and lighter to run than MediaWiki. The old wiki will remain accessible at its old address (as read-only).

I quite enjoyed the chance to do some web development. The new implementation aims to be as light-weight as possible: there are no frameworks or toolkits in use beyond basic PHP and MySQL. Shared elements like the top bar are easily inserted into other pages via PHP includes. File caching is used in many places to reduce database access and other repeated processing. I’ve also been paying attention to making the CSS responsive and mobile-friendly.

Let me know if you find anything that seems broken. I will undoubtedly continue doing minor tweaks to the site over the coming weeks.

Memory savings and metadata cache

Build 2237 introduces a number of changes aimed at making Doomsday launch faster and use less memory overall. It also fixes a couple of threading issues that were found with the help of Thread Sanitizer.

As part of preparing the stable 2.0 release, I’ve been looking at performance and memory optimizations.

Trimming memory use

I examined memory use in Instruments on macOS and identified a number of places for savings:

  • There is now a purpose-built class for keeping track of observer relationships between objects. There are hundreds of thousands of these being used in the engine, so using a class that is more optimal for the task saves both memory and improves performance.
  • Some objects had a lot of observers (20,000+) but worst of all, this was completely unnecessary because the observers were never being notified. These cases have now been removed.
  • Reduced memory used by Doomsday Script bindings for file system access by removing several unnecessary variables and replacing them with callable functions. This saves memory because functions can be stored in the class objects rather than in the objects themselves.
  • The file system caches uncompressed ZIP file contents in memory to avoid repeatedly decompressing them when the ZIP is being accessed. However, these cached contents were never released. Now ZIP caches are freed from memory whenever you return back to the Home screen.

After all these changes, Doomsday’s memory use on my system was roughly halved (from 830 MB to 460 MB). In practice, though, this heavily depends on how many resource packs and other data files you have available.

Caching metadata

When Doomsday is starting up it indexes and quickly analyzes all the resource packs, add-ons and other data files that you may have available. This may take a while depending on the number of these files and how fast your hard drive is. Starting with build 2237, Doomsday is now able to cache this information for future use, so after the analysis has been done it doesn’t have to repeated until the files change or new ones are added. Continue reading Memory savings and metadata cache

Improved UI for multiplayer

For the past two weeks I’ve been focusing on finishing the Home UI for 2.0. I’ve also been doing plenty of small improvements and bug fixes.

The final element that has been missing from the Home is the Multiplayer tab. That is now no longer the case: build 2227 has a new UI for viewing server information, and a couple of important bug fixes for multiplayer servers.

Server info dialog

In the Multiplayer tab you can see more information about a server by right-clicking it or by clicking the […] button in the server description. This pops up the new dialog that can be seen in the screenshot above.

A direct connection to the server is automatically attempted so that you’ll see the server’s current status and to verify that the server is reachable. The status information includes the current map outline, just like in the Shell.

In addition to status information, the dialog has a “Join Game” button, a list of the packages loaded on the server (the client must load the exact same ones when it connects), and you can choose additional packages to load locally. These local packages are game-specific rather than server-specific and are remembered in the client configuration (part of persist.pack). The “Local Packages” button is disabled by default. It can be enabled in Network Settings.

Note that if you load additional local packages in multiplayer games, you must ensure that they don’t contain gameplay-affecting changes. For example, loading a different map is out of the question, as are changes to mobj states. Doomsday 2 packages with 3D model assets are safe to use locally in multiplayer games.

I now consider this UI, along with the rest of the Home, feature-complete for 2.0. The long-term thinking behind the server info dialog is to evolve it toward a mini-Shell interface so that you could issue quick console commands to servers that you control. That will not be part of the 2.0 release, though.

Continue reading Improved UI for multiplayer

Improved UI for packages

For the past week I’ve been finishing up with the Home UI for the 2.0 release. The Packages tab has gained a lot of useful functionality.

Package popup

The Packages tab helps you browse and manage your PWADs, add-ons, resource packs and other data files (all of which are called “packages” in Doomsday 2). Build 2208 introduces a new UI for viewing information about packages and performing quick actions on them. Like the game tabs, I consider the Packages tab now feature-complete for 2.0.

The new package info popup has the following features:

  • The popup can be quickly opened by right-clicking a package in any Packages list.
  • There is a convenient “Play in…” button for quickly loading the package in an existing game profile, without making any permanent changes to profiles. This works great for quickly trying out PWADs, for example.
  • The “Add to…” button lets you add the package to any existing game profile.
  • The “Show File” button opens the folder containing the package in Windows Explorer / macOS Finder / etc. (This may not be very helpful if all your data files are lumped in a single folder.)
  • The “Options” button allows selecting from the optional .box contents. It was relocated here from the old context menu that no longer exists.

A little bit of content analysis is done on the package:

  • Doomsday tries to guess which game the package is for, looking through any auxiliary readme files, the used WAD map format, Snowberry metadata, names of parent folder(s), etc.
  • In the case of WAD files, the game title picture is shown in the popup as the package icon.
  • If the package contains an “icon.jpg” (or “icon.png”) file, it is used as the package icon image. Currently these images are only displayed in the popup.

Continue reading Improved UI for packages