Some time ago I wrote that IBM OS/2 1.0 and 1.1 cannot be installed in a VM due to the way it switches between real and protected mode. At the time I did not realize that there was another obstacle, namely IBM’s use of the undocumented 80286 LOADALL instruction.

For reasons that are not very clear, Microsoft’s editions of OS/2 used 386 instructions since before OS/2 1.0 was released, but IBM’s versions did not. When Microsoft’s OS/2 kernels detected that they were running on an 80386 (or newer) processor, they used 386-specific code to return from protected mode back to real mode. This was faster than the 286 method which required a CPU reset (OS/2 did not utilize the common method which relied on the PC/AT keyboard controller to reset the CPU and instead caused a processor shutdown through a triple fault, inducing a reset that way; the OS/2 method ought to be faster). Microsoft’s OS/2 kernels also didn’t use the LOADALL instruction when running on a 386.

IBM’s OS/2 1.0/1.1 kernels on the other hand didn’t care one bit about running on a 386, at least not on an AT-compatible machine—perhaps because at the time IBM had no AT-compatible 386 systems on the market or even in the pipeline. IBM’s OS/2 1.0 can still run on a 386 or newer CPU, but it requires a bit of help from the machine’s BIOS.

The BIOS must properly support software that sets the CMOS shutdown status byte, resets the CPU (either by triple-faulting, through the keyboard controller, or by any other means), and regains control after the reset while minimally disturbing the machine state. That functionality is common enough because it was also used by DOS extenders and many OS boot loaders.

What’s less common is BIOS support for emulating a 286 LOADALL instruction. Even though the 80386 has its own undocumented LOADALL, a 486 or later CPU has nothing of the sort. However, a 386 or later can emulate enough of 286 LOADALL functionality to satisfy common uses of it, which includes things like old HIMEM.SYS, RAMDRIVE.SYS, or OS/2.

The VirtualBox BIOS has had support for LOADALL emulation for a while now, so it should be able to run IBM’s OS/2 1.0. And it is able to run it… but not install. Or not quite. Fortunately there’s a way around it…

When using the VirtualBox defaults for OS/2 1.x, a 500 MB IDE drive is chosen. OS/2 1.0 can work with that, but the FORMAT utility shipped with IBM OS/2 1.0 can not. The OS/2 installer can create a default partition (limited to FAT-12!) just fine, but trying to format it results in the following error:

“FORMAT unsuccessful” is not exactly self-explanatory. And good luck getting help from your dealer or the IBM representative that sold you IBM OS/2.

The INSTALL.LOG file is supposed to show the “condition that caused the error”, but really, it does not:

In other words… the FORMAT utility failed, but the OS does not have the faintest clue what the problem might be.

After I failed to find any problems with disk access, I began suspecting that the old FORMAT might not like “modern” drive geometry. That would not be a first exactly, for example old DOS versions have a known bug that may cause them to fail on a hard disk with more than the then-common 17 sectors per track.

I was able to confirm that this is indeed the problem. The FORMAT utility in OS/2 1.0 works with 17 sectors per track common on MFM hard disks, and it also works with 26 sectors per track common on RLL hard disks. It even works with 40 sectors per track, but not 50 or more.

There are two ways around this in VirtualBox. The complicated one is creating a VHD or VMDK disk image with appropriate physical geometry and letting OS/2 format the disk. The significantly easier workaround is letting OS/2 create a partition and fail during format, then use DOS to format the partition; once the partition is formatted by DOS, the OS/2 installer can be restarted, told to not format the C: drive, and happily continue.

Again, the problem is only in the FORMAT utility shipped with IBM’s OS/2 1.0, the operating system as such has no trouble with IDE disks that use 16 heads and 63 sectors per track.

The steps to install IBM OS/2 1.0 in VirtualBox (version 6.1.x or later ought to work) are as follows:

1. Create a VM using the defaults for OS/2 1.x
2. If not already done, patch the installation floppy image to avoid division by zero crashes
3. Start the installer, let it create a partition, and fail during format
4. Use more or less any DOS version to format the C: drive
5. Restart the OS/2 installer, do not format the partition, and install the OS

Mouse Extras

For extra points, it is possible to get working mouse support. IBM OS/2 1.0 does not ship with a driver for PS/2 mice on PC/AT compatible systems. But it is possible to borrow one from a Microsoft OS/2 edition which supports the combination. One possible donor is Nokia OS/2 1.1.

To accomplish this, it’s easiest to install OS/2 1.0 with more or less any mouse driver. It won’t work, but will set up the CONFIG.SYS entries. Then insert Disk 1 from Nokia OS/2 1.0, start an OS/2 command prompt, switch to the C:\OS2 directory, and run the following command:

a:\unpack a:\mousea05.sy@

This will copy and uncompress the MOUSEA05.SYS driver. After that, it is necessary to either modify CONFIG.SYS to use MOUSEA05.SYS instead of the original driver, or copy MOUSEA05.SYS over the driver set up by the OS/2 installer.

Those unable or unwilling to use EDLIN (the only text editor shipped with OS/2 1.0) can boot a DOS floppy and use any DOS editor to fix up OS/2’s CONFIG.SYS.

When everything is done, OS/2 should reboot to the this screen:

Admittedly this is not quite as straightforward as installing IBM OS/2 1.0 in an emulator that mimics an IBM PC/AT. But it’s doable, and the ancient OS runs blazingly fast on a modern computer.