Gotta Go Fast: Building old Windows on new Windows

The Windows XP SP1 and Server 2003 source code leaked recently, and it includes the build system. While it isn’t exactly simple to get it up and running, and not everything is included (missing winlogon is the biggest problem), people have already figured it out and managed to make working VMs from it.

A quick disclaimer: Nothing in this blog post contains any source or copyrighted material from the leak, in any form. Don’t ask me for the leaked source, and don’t ask me where to find the leaked source, because I won’t provide that information. Everything here is provided for educational purposes, since there is a lot of value in being able to get old undocumented build processes working on modern computers. If anyone at Microsoft has any concerns about the content of this blog post, please feel free contact me. My email address is my twitter handle at gmail.

I’ve been playing around with the build system, for a laugh, but found that it is painfully slow to build. One of the reasons is that, so far, the only way to build it has been on a Windows XP or 2003 VM. I found that, at least under Hyper-V on my current hardware, Windows XP is not very happy. At idle it munches through about 50% of the CPU time on 16 host cores. Running compiles from a VM also tends to suck because of the additional IO overhead. Plus the build tooling was designed in a time where cores were not a thing. Yes, you got HyperThreading on server processors, but if you wanted more than one actual processor you bought a multi-socket motherboard.

You can technically build the Windows Server 2003 source from Windows 10, but it isn’t as simple. For a start you need to run a 32-bit OS, because you need NTVDM to run the 16-bit build tooling. I don’t know anyone (aside from maybe Foone, who is just like that) who would willingly run a 32-bit version of Windows 10 in 2020. I certainly don’t. It also does not like being run on a modern 64-bit build system in general, because it tries to use the AMD64 tooling even for 32-bit build targets, and you definitely do not have the right Microsoft Visual C++ Runtimes installed, because the installers don’t work on Windows 10.

So how do you get around this? I’ve figured most of it out, but I’m still working on the 16-bit tooling stuff. Here’s the process I got working:

1. Install the Windows Server 2003 SP1 Platform SDK.
2. Copy all files matching msv*.dll (aside from msvideo) from a Windows XP SP1 or Windows Server 2003 SP1 installation, and place them into the win2k3\nt\tools\x86 directory. These are the MSVC++ runtime binaries.
3. Launch a command prompt as administrator. This is necessary because the build process messes with the ACLs on all the build files, and if you’re not admin it becomes a huge pain in the ass.
4. Change directory to C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Microsoft Platform SDK for Windows Server 2003 SP1\Open Build Environment Window\Windows Server 2003 32-bit Build Environment.
5. Execute Set Win Svr 2003 32-bit Build Env (Retail).lnk, which sets up the path and environment variables to do a build.
6. Change directory to win2k3\nt\tools and run razzle no_certcheck Win32. This does all the build setup magic for an x86 target.
7. The new ACLs break a bunch of stuff, so change directory back into win2k3\nt\tools\x86 and run icacls *.dll /grant Everyone:(OI)(CI)RX to grant read+exec permissions to everyone. I also found that I needed to do some manual permissions pruning later – you may just want to grant blanket full permissions to save messing around.
8. Run echo %PATH% to see the current path. It will almost certainly start with an entry ending in AMD64. Copy the path without that first part, and run set PATH=... with the new path, so that the build process won’t try to use the AMD64 tooling.
9. Repeat the last step again for %BUILD_PATH%.
10. Run the build.

I won’t explain the build process in any further detail. Partly because it’s clear that Microsoft are feeling particularly litigious about it, but mostly because this blog post isn’t intended to be a tutorial on how to build leaked source code.

The next problem, if you have a machine with lots of cores, is that the build.exe tool won’t support more than 32 parallel jobs. One half of this restriction is that the -M command line parameter is hard-coded to not allow more than 32 jobs. The other half is that, if no specific degree of parallelism is provided, it uses one job per CPU as detected by the GetSystemInfo API. If you have more than 32 logical processors on your system (i.e. both normal and hyperthreading cores) the API will just return 32, because a 32-bit process is limited to 32 processors in its processor group, due to the size of the internal KAFFINITY value in the 32-bit kernel (which is emulated by WOW64 for 32-bit processes on a 64-bit system). If the build tool were actually using threads for task parallelism, this would be a hard limitation, because the build process couldn’t schedule its threads on any processors outside of that group of 32 anyway (at least without manual intervention). But the build process actually spawns new processes for task parallelism, which the OS will distribute cleanly across available processor groups, even though the 32-bit processes can’t see that happening because of the WOW64 translation layer.

To get around this limit, I wrote BuildFaster, which patches the build tool to accept -M values up to 127, as well as override the job count globally if the BUILD_THREADS environment variable is set. To be absolutely clear: the patcher does not contain any part of the build.exe tool, nor any other copyrighted materials – it just writes some new bytes into a copy of the file that the user provides.

I’ve been using a job count of around 200 on a system with 112 logical processors and a couple hundred gigs of RAM, with all the build files on an NVMe SSD array. Build times are now in the order of minutes, compared to the 1-2 hours it took on a Windows XP VM.

At the moment I can get most of the build to complete. All of the C/C++ compiles, as does the MIDL stuff. The final issue is the 16-bit tooling, which won’t run on a 64-bit machine because NTVDM isn’t supported. I’m looking into NTVDMx64 as a potential solution, but it’s non-trivial.