**Lithtech Jupiter – Building on Windows 10** Intro ======================================================================================= I got my hands on Lithtech Jupiter. Monolith’s (or rather, Touchdown Entertainment) engine used for games like No One Lives Forever 2. The source is from the early 2000’s and comes with VS 2003.NET solutions. Lithtech Jupiter’s development kit is out there and several GitHub repositories have been sharing the code for quite some time. Compiling it on modern C++ compilers, however, isn’t possible without some modifications. This article documents my journey getting Lithtech to compile and run on Windows 10 with Visual Studio 2017. It mostly should serve myself to keep track of what I did, but since I couldn’t get any of the existing code in the mentioned GitHub repositories to compile out of the box, it might be helpful and interesting to someone else as well. Roughly speaking, Lithtech is a Client-Server architecture. The server resides in the main engine executable. If the program runs locally, localhost is the server. The engine is self-contained and doesn’t know about any game code. The engine just exposes an interface to the outside so that programmers can implement the game code without worrying about engine implementation details. The game code is compiled as a DLL and is loaded dynamically when Lithtech.exe (the core engine) starts. Project overview ======================================================================================= The development environment is organized in several directories of which the following three are of particular interest: Dir | Purpose -------------------------------------------|--------------------------------------------- `/Engine` | Engine Source Code `/Game` | Game Source Code `/Development/< Game Name >` | All of the games’ data eg. sounds, models, maps, etc. When building the engine, a post-build step (a win batch command) will copy the engine Executable (lithtech.exe) and necessary DLLs to the /Development/ directory. The game sourcecode will get compiled into a DLL (along several other DLLs). Those also get copied to /Development//Game after the build has finished. The following system variables have to be defined, so that the copy commands (and the development tools like the World Editor) know where the game resides: Var | Content -------------------------------------------|--------------------------------------------- `GAME_MAIN_DIR` | GAME_MAIN_DIR: Root directory of the game folder, eg: | G:\TouchdownEntertainment\Jupiter_Ent\Development\TO2 `GAME_REZ_DIR` | Where the game data is located (Props, Music, models, worlds,… ), eg. | G:\TouchdownEntertainmentCopy\Jupiter_Ent\Development\TO2\Game `GAME_TOOLS_DIR` | Contains development tools, such es Dedit (level editor) and so on. Set this to eg. | G:\TouchdownEntertainmentCopy\Jupiter_Ent\Tools “TO2” is the standard sample game folder that comes with the Lithtech Jupiter engine environment. It contains the data of the first section of the first level of the game “No One Lives Forever 2”. Since the first NOLF was actually called: “The Operative: No One Lives Forever”, it is save to assume that “TO2” just stands for “The Operative 2”. The Engine solution contains several projects. One of them builds the Executable “Lithtech.exe”. It depends on several libraries:  As you can see, the Direct3D renderer for example, is statically linked right into the engine. Building the libraries & Lithtech.EXE ======================================================================================= Firstly I tried to compile each of those static libraries individually. In the Code generation section of the projects’ properties I changed the runtime to /MDd as some of them were linking to the static version /MTd. As far as I know you cannot mix the static and dynamic version of the runtime. I don’t know why this setting is inconsistent for the projects but the original solution file was created in VS2003.NET so some things might have gone wrong during the conversion to VS2017? Not sure. Also I set the build config to “Debug” and not “Debug with MFC”, as the latter will set the EXE_LithTech optimization to /O2 which I do not want for the moment. I don’t want any variables optimized away when I get to the part where stepping through the code with the debugger is possible. LIB_D3DRender --------------------------------------------------------------------------------------- The Renderer uses Direct3D 9. Ususally the DX9 API does not come with Win10 and I also was not able to install it using the VS installer. I was able to find the SDK, though, and install it onto my harddrive. The project, of course, doesn’t know about the SDK yet:  Adding the include folder from the DirectX9 SDK to additional include directories, fixes this. Of couse, in the linker settings the path to the D3D9 import libraries has to be set as well. I just copied the SDK to the engines folder so there are no absolute paths. With the missing includes and libraries out of the way the compiler complains about a ton of errors of this kind:  Looking at the source, they are caused by undefined variables. Apparently, in early C++, variables defined within a loop “leaked” out of the loop’s scope:  Simply defining the variable before the loop fixes this. There are many of those errors and I had to go through them one by one and pull the out of the loop or redefine them when they get reset in a following loop. LIB_DShow --------------------------------------------------------------------------------------- In the Configuration Properties → C/C++ → Preprocessor → Preprocessor Definitions we find WINVER=0x500. Looking at https://docs.microsoft.com/de-de/cpp/porting/modifying-winver-and-win32-winnt?view=msvc-160 this stands for Windows 2000. This will cause the compiler to complain about missing LCMapStringEx which is part of the MFC (Microsoft Foundation Classes), since the function’s signature has changed. Changing 0x500 to 0x0A00 solved this problem. Next, we have the assignment operator overloaded without specifying the return type:  Probably that was just fine back then. Actually, I was not able to find the operators implementation anywhere in the code. There is an implementation of this operator in COARefTime which takes a double and as the comment says there might be a bug when a LONG is passed to it instead of a double. So this might be some C++ trick to fool the compiler, I am not sure. I commented those out and added a TODO. This might bite us later. Next up, there is a case of default integer initialization. Not going to fly on modern C++ compilers either. In this case it is in wxdebug.cpp: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C static g_dwLastRefresh = 0; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Setting this explicitly to DWORD solves the issue. DWORD because it is later subtracted from a variable which is of type DWORD as well. Finally, there are a bunch of those variables the compiler doesn’t know what to do with, because their initialization happens within a for-loop just like for the d3d renderer earlier. Fixing those let me compile the library just fine. LIB_Info --------------------------------------------------------------------------------------- This one complains about an _assert identifier that cannot be found:  Usually I see this type of assert being implemented by the programmer herself, as only assert(expression) is available through the standard library via assert.h. There is an _assert but it's not supposed to be called by the user, as the documentation states: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/assert-macro-assert-wassert?view=msvc-160 LIB_RezMgr --------------------------------------------------------------------------------------- Another one with an undefined variable, because of its definition happening inside of a loop. Also, the identifier “toupper” cannot be found in rezutils.cpp. Including ctype.h solves this problem. LIB_UI --------------------------------------------------------------------------------------- Lib_UI: another “default typed” constant: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C const kCharSpacing = 2; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I changed this to an int as it is used as such further down in the code. Compiles after this change. Lib_Lith, Lib_NullRender, Lib_ILTSound, Lib_StdLith, Lib_Zlib, Itmem --------------------------------------------------------------------------------------- All of those compile without any modifications. Sweet! EXE_LithTech --------------------------------------------------------------------------------------- Finally, we have all the stuff together to compile the Engine runtime. The compiler will cry about the missing header for dmusici.h. But we just add the DX9 SDK to the include directories and all is good. By the way, I put the DX9 SDK under G:\TouchdownEntertainment\Jupiter_Ent\Engine\runtime\DX9\DX9Include. Next up, a ton of those undefined variables popped up because of the “loop leakage” as seen before. The following error was more interesting and, honestly speaking, I don’t know yet how to handle it properly. In socket.h there is a redefinition of identifiers that are #defined in errno.h, which is part of the WinSDK:  The thing is, I don’t know if the values of WSAEWOULDBLOCK and WSAECONNRESET have changed since Visual Studio 2003.NET and, more importantly, if that is relevant to the engine when it uses EWOULDBLOCK and ECONNRESET. I don’t think so but I am also not entirely sure. I redefined the two problematic constants to: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C const int MY_EWOULDBLOCK = WSAEWOULDBLOCK; const int MY_ECONNRESET = WSAECONNRESET; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ and replaced the occurrences of EWOULDBLOCK and ECONNRESET with those in the following code. I made a big TODO here and I will come back to it later. This approach might seem hastly but when you have to deal with a fairly large codebase that doesn’t compile at all I believe it is better to defer certain things and not get hung up on the details. Later, when the code compiles and runs we can revisit the parts we marked as potentially problematic. ### DirectX 9 This project also will need the DirectX 9 input libraries. Adding their path to the Additional Library Directories does the job. ### ATL ATL.lib input file cannot be found. Another linker issue. ATL stands for Active Template Library and is a lightweight form of the MFC, according to wikipedia: https://en.wikipedia.org/wiki/Active_Template_Library It also states that one cannot link dynamically to ATL anymore. That is probably why atl.lib cannot be found. I believe atl.lib is the import library responsible for loading the DLL with the actual code. ATLS.lib contains the static version of that thing so in the libraries settings change it from atl.lib to atls.lib. After this, the linker complains: CfactoryTemplate: unresolved external symbol "class CFactoryTemplate * g_Templates" (?g_Templates@@3PAVCFactoryTemplate@@A) Same problem with g_cTemplates. Those are declared in dllentry.cpp: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C extern CFactoryTemplate g_Templates[]; extern int g_cTemplates; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Definition seems to happen in sysclock.cpp: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C #ifdef FILTER_DLL /* List of class IDs and creator functions for the class factory. This provides the link between the OLE entry point in the DLL and an object being created. The class factory will call the static CreateInstance function when it is asked to create a CLSID_SystemClock object */ CFactoryTemplate g_Templates[1] = { {&CLSID_SystemClock, CSystemClock::CreateInstance} }; int g_cTemplates = sizeof(g_Templates) / sizeof(g_Templates[0]); #endif ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ However, FILTER_DLL is undefined. If I define it the compiler complains about type mismatch of {&CLSID_SystemClock, CsystemClock::CreateInstance}. This whole thing seems to follow a pattern described here: https://docs.microsoft.com/en-us/windows/win32/directshow/class-factories-and-factory-templates. I am not familiar with DirectShow but apparently the programmer would define Filters for, say, a video-stream, which modifies the stream in some way. Many filters could be chained together to transform data. At least this gives us a hint, that apparently only one of those filters is being created (if it is even created at all). That way, for now I define g_Templates and g_cTemplates as follows: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C CFactoryTemplate g_Templates[1]; int g_cTemplates = 0; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The reason I set g_cTemplates to 0 is because g_Templates is never really initialized. So we only do this to keep the linker happy. Also, if at some point FILTER_DLL is enabled those should be set to extern again without the initialization. Otherwise we have multiple definitions. So we will see if this will work at all later on. I will leave a comment at this point that we do not actually build the filter. Now the executable Lithtech.exe is being built. Yes!!! ### One more thing A Post-Build command is configured as: copy "$(TargetPath)" "$(GAME_MAIN_DIR)" This is where the system variable comes in. $(TargetPath) is set to: $(ProjectName) ( Right click project → Properties → Configuration Properties → General → Target Name ). We set this to “Lithtech”. Otherwise the copy command will try to copy Exe_Lithtech.exe which doesn’t exist. I was not able to find the variable name for the output file the linker generates. It is very hard to find what you need in the online doc about VS. Just sayin’. Iterator Invalidation ======================================================================================= Both engine and Game code suffer from iterator invalidation. Example from ScatterFX.cpp in function UpdateSubVolume: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C // update which subvolumes are visible and setup LOD bool CScatterFX::UpdateSubVolumes( void ) { // make sure there are no active subvolumes if scatter isn't enabled if( !m_bEnabled ) { std::set::iterator it = activeSubVolumes.begin(); for( ; it != activeSubVolumes.end(); it++ ) { (*it)->Deactivate(); } return true; } … } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is not obvious at first glance but the problem is in Deactivate(): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C bool CScatterFXSubVolume::Deactivate( void ) { if( !m_Active ) return true; g_pCommonLT->SetObjectFlags( m_Effect, OFT_Flags, 0, FLAG_VISIBLE ); m_Parent->freeEffects.push_front( m_Effect ); m_Effect = NULL; m_Active = false; m_Parent->activeSubVolumes.erase( this ); return true; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ m_Parent holds a reference to the CspecialFX instance we called Deactivate from. So when erase is called on m_Parent→activeSubvolumes we modify the very set we got the iterator from. Therefore the Iterator is invalidated and the program will probably crash when it tries to compare it next time to activeSubVolumes.end(). A crash is actually the prefered thing. Strictly speaking it doesn’t have to. It is – to my knowledge – implementation dependent what happens when you’re playing with invalidated iterators. No good! A fix: Firstly, we do not erase the element from the set in Deactivate and secondly we change the return behavior: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C bool CScatterFXSubVolume::Deactivate( void ) { if( !m_Active ) return false; g_pCommonLT->SetObjectFlags( m_Effect, OFT_Flags, 0, FLAG_VISIBLE ); m_Parent->freeEffects.push_front( m_Effect ); m_Effect = NULL; m_Active = false; return true; } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ and in the calling loop we do this: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C std::set::iterator it = activeSubVolumes.begin(); for( ; it != activeSubVolumes.end(); ) { if ( (*it)->Deactivate() ) { it = activeSubVolumes.erase(it); } else { ++it; } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Only if the sub volume is active we do the things we need to do in Deactivate and then erase the subvolume from activeSubVolumes and update the iterator with the element one past the erased element. That way the iterator stays valid. Otherwise nothing should be done and the iterator is incremented by one element. I checked the code for the appearances of Deactivate so it won’t conflict with the changed behavior and it seems to be fine. The return types aren’t actually being used anywhere so at least now we can make good use of them. Deactivate is also called further down the UpdateSubVolumes function: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C // activate subvolumes with positive LOD for( uint32 i = 0; i < m_nNumSubVolumes; i++ ) { float curLOD = m_pSubVolumes[i].CalculateLOD( m_vCamPos ); if( curLOD > 0.0f ) m_pSubVolumes[i].Activate( curLOD ); else if (m_pSubVolumes[i].IsActive()) { m_pSubVolumes[i].Deactivate(); activeSubVolumes.erase( m_pSubVolumes + i ); // Added this } } ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We now have to call erase explicitly to remove it from the active subvolumes set. Actually, when erasing an object by its reference from a set I noticed that the CPU crunches through a TON of instructions. I wonder how good this is. But I also noticed that this function is called sparingly, not every frame. But still… The pattern of a Class holding an array of things (in this case CscatterFXSubVolume) along with its total count and a set that holds the “active” references of the array is used throughout the game’s codebase. An excerpt of the CscatterFX class: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C uint32 m_nNumSubVolumes; // number of subvolumes in this volume CScatterFXSubVolume* m_pSubVolumes; // array of subvolumes within this scatter volume // shared list of volumeeffect objects that have been allocated, but aren't currently used static std::list freeEffects; // collection of subvolumes that are currently active std::set activeSubVolumes; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ So there are more of potentially invalid iterators to fix for sure.