OpenBSD Guest with Bhyve on OmniOS

Today I will be creating a OpenBSD guest via bhyve on OmniOS . I will also be adding a Pass Through Ethernet Controller so I can have a multi-homed guest that will serve as a firewall/router.

This post will cover setting up bhyve on OmniOS , so it will also be a good introduction to bhyve . As well, I look into OpenBSD’s uEFI boot loader so if you have had trouble with this, then you are in the right place.

Install bhyve

I will cover installing bhyve onto an OmniOS system briefly here. If you already have bhyve installed, you can jump ahead toSetting up the OpenBSD boot disk.

To install bhyve onto OmniOS , enter the command below:

pkg install brand/bhyve

This will also install system/bhyve . Simple!

Create storage for the OmniOS zone guests

Every zone on a system has at least one dedicated ZFS dataset and it’s good practice to create a parent dataset under where all zones will live. A common convention is to mount this dataset as /zones and create it at the top level of a pool.

I will create a zpool named data , on disk device c2t1d0

# zpool create data c2t1d0

This can be verified with the following command:

# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
data    928G   526K   928G        -         -     0%     0%  1.00x  ONLINE  -
rpool   232G  7.98G   224G        -         -     0%     3%  1.00x  ONLINE  -

Now that this zpool exists, the following command will create the top-level ZFS dataset , named zones :

# zfs create -o mountpoint=/zones data/zones

The creation of the ZFS dataset can be verified with the following command:

# zfs list
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
data                                 153K   899G    24K  /data
data/zones                            24K   899G    24K  /zones
rpool                               15.4G   209G    26K  /rpool
rpool/ROOT                          1001M   209G    24K  legacy
rpool/ROOT/omnios-r151034           44.5M   209G   620M  legacy
...

That is all there is to setting up your bhyve environment!

Setting up the OpenBSD boot disk

Download OpenBSD miniroot67.fs disk image:

# mkdir /zones/iso
# cd /zones/iso
# wget https://cdn.openbsd.org/pub/OpenBSD/6.7/amd64/miniroot67.fs

Create a block device to boot from:

# lofiadm -a /zones/iso/miniroot67.fs 
/dev/lofi/1

Create a virtual network interface card ( vnic )

Create a vnic with the following command:

# dladm create-vnic -l e1000g0 openbsd0

Note: I use the hostname as the zone name for the vnic .

The new vnic can be verifed with the following command:

# dladm show-link
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       --         --
openbsd0    vnic      1500   up       --         e1000g0

Configure the physical network card for PCI-Passthru

First search prtconf to find the device we will be adding to OpenBSD:

# prtconf -dD
...
...
pci8086,8c18 (pciex8086,8c18) [Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5], instance #0 (driver name: pcieb)
            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #1 (driver name: igb)
            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #2 (driver name: igb)
            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #3 (driver name: igb)
            pci8086,1 (pciex8086,1521) [Intel Corporation I350 Gigabit Network Connection], instance #4 (driver name: igb)
...
...

I will be adding the I350 Gigabit Network device, found and listed above in the output from prtconf -dD .

We need to convert this into a ppt device and this can be done by adding the PCI Identifiers to the ppt_aliases & ppt_matches files, as follows:

# echo 'ppt "pciex8086,1521"' >> /etc/ppt_aliases
# echo 'pciex8086,1521' >> /etc/ppt_matches

For the system to attach these as ppt devices a reboot is necessary. Once your system has come back online, you can verify that the ppt devices are attached:

# pptadm list -a
DEV        VENDOR DEVICE PATH
/dev/ppt1  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0
/dev/ppt2  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,1
/dev/ppt3  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,2
/dev/ppt4  8086   1521   /pci@0,0/pci8086,8c18@1c,4/pci8086,1@0,3

Now these devices are ready to be passed through to bhyve

Create the ZFS dataset to store the bootdisk for the guest

This is the ZFS dataset that will be the storage device for the OpenBSD bhyve zone. Create the ZFS dataset as follows:

omnios# zfs create -V 10G data/openbsd

Configure the OpenBSD bhyve zone

Zones are configured on OmniOS via the zonecfg command, this is also true for bhyve zones.

The following can be pasted into zonecfg once it has been executed. You will need to change any directives that are specific to your system.

# zonecfg -z openbsd
openbsd: No such zone configured
Use 'create' to begin configuring a new zone.

create -b
set brand=bhyve
add attr
    set name=type
    set type=string
    set value=openbsd
end
set zonepath=/zones/openbsd
set ip-type=exclusive
add net
    set allowed-address=192.168.1.67/24
    set physical=openbsd0
end
add device
    set match=/dev/lofi/1
end
add attr
    set name=disk
    set type=string
    set value=/dev/lofi/1
end
add device
    set match=/dev/zvol/rdsk/data/openbsd
end
add attr
    set name=bootdisk
    set type=string
    set value=/dev/zvol/rdsk/data/openbsd
end
add attr
    set name=acpi
    set type=string
    set value=off
end
add attr
    set name=bootrom
    set type=string
    set value=BHYVE_RELEASE
end
add device
    set match=/dev/ppt1
end
add attr
  set name=extra
  set type=string
  set value="-S -s 8:0,passthru,/dev/ppt1"
end
verify
commit
exit

The above is quite intuitive and should not need any explanation.

Install the OpenBSD bhyve zone

Once a new zone has been configured with zonecfg it needs to be installed. This is achieved with the following command:

# zoneadm -z openbsd install

Boot the OpenBSD bhyve zone

Everything is now set and we can boot the OpenBSD bhyve zone as follows:

# zoneadm -z openbsd boot

You now have 5 seconds to access the OpenBSD boot loader. After this time, you will not be able to access the OpenBSD bhyve zone via the console. Or at all… that bird will have flown and you will have to reboot the zone.

Enter the console on the OpenBSD bhyve zone

Configuration of the OpenBSD guest needs to be done via the console, as follows:

# zlogin -C openbsd
[Connected to zone 'openbsd' console]

Immeadiatleyafter entering the console, press the space bar & then the enter key to stop OpenBSD booting into the VGA Output mode.

Setting the console in the OpenBSD bhyve zone

After successfully accessing the console, the boot> prompt will appear. The OpenBSD guest now needs to be configured to redirect all output to the com0 serial port.

boot> set tty com0

Boot the OpenBSD bhyve zone

Once the com0 serial console is set, the OpenBSD guest can be booted:

boot> boot

OpenBSD can now be installed as normal. The Pass-Thru Ethernet Device will show in the network configuration as a normal device and no special care needs to be taken.

Where it gets tricky is the Disk Creation section, which I cover next

Setting up the root disk with uEFI

For me, GPT is the sure-fire way to make sure uEFI is installed… all the time… every time. You will see a warning about uEFI & GPT , but from my experiences, this can safley be ignored. And besides, we will do a full check after installation just to make sure uEFI is where it is supposed to be.

So when you reach that part of the installation that deals with the disk, we want to be installing onto sd0 and the creation looks a little something like this:

Available disks are: sd0 sd1.
Which disk is the root disk? ('?' for details) [sd0] sd0
No valid MBR or GPT.
Use (W)hole disk MBR, whole disk (G)PT or (E)dit? [whole] G
An EFI/GPT disk may not boot. Proceed? [no] yes
Setting OpenBSD GPT partition to whole sd0...done.
The auto-allocated layout for sd0 is:
#                size           offset  fstype [fsize bsize   cpg]
  a:       1218736.0K             1024  4.2BSD   2048 16384     1 # /
  b:        262144.0K          2438496    swap                    
  c:      10485760.0K                0  unused                    
  d:       3145728.0K          2962784  4.2BSD   2048 16384     1 # /usr
  e:       2097152.0K          9254240  4.2BSD   2048 16384     1 # /home
  i:           480.0K               64   MSDOS                    
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a]
/dev/rsd0a: 1190.2MB in 2437472 sectors of 512 bytes
6 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd0e: 2048.0MB in 4194304 sectors of 512 bytes
11 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
/dev/rsd0d: 3072.0MB in 6291456 sectors of 512 bytes
16 cylinder groups of 202.50MB, 12960 blocks, 25920 inodes each
Available disks are: sd0.
Which disk do you wish to initialize? (or 'done') [done] 
/dev/sd0a (a3b707add095aa86.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0e (a3b707add095aa86.e) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (a3b707add095aa86.d) on /mnt/usr type ffs (rw, asynchronous, local, nodev)

Continue on with installing the file sets via http and when that is all finished, we will make those checks for the uEFI boot loader.

Checking the root disk for uEFI

Exit out of the installer and into the shell and from here we can verify all the right uEFI bits are in place:

CONGRATULATIONS! Your OpenBSD install has been successfully completed!

When you login to your new system the first time, please read your mail
using the 'mail' command.

Exit to (S)hell, (H)alt or (R)eboot? [reboot] S
To boot the new system, enter 'reboot' at the command prompt.

openbsd# fdisk sd0
Disk: sd0       Usable LBA: 64 to 20971456 [20971520 Sectors]
   #: type                                 [       start:         size ]
------------------------------------------------------------------------
   1: EFI Sys                              [          64:          960 ]
   3: OpenBSD                              [        1024:     20970433 ]

To be absolutely certain, we can check inside the uEFI partiton to make sure the bootx64.efi file is available.

openbsd# mount /dev/sd0i /mnt2                                                 
openbsd# ls /mnt2/efi/boot/     
bootia32.efi bootx64.efi
openbsd# umount /mnt2

Looking good!

Reboot the system

We can issue the reboot command and our new OpenBSD bhyve zone will come up cleanly:

# reboot

>> OpenBSD/amd64 BOOTX64 3.50
boot> 
booting hd0a:/bsd: 12879176+2753552+324816+0+704512 [813343+128+1022112+747816]=0x125cf68
entry point at 0x1001000
[ using 2584424 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.7 (GENERIC) #179: Thu May  7 11:02:37 MDT 2020
    [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1056288768 (1007MB)
...
...
...
...

Escaping from the console

To escape from the console, use ~~. You will be brought back to the OmniOS system.

Post installation tasks

There is nothing necessary to be done, however a little tidying up is always good.

This should be done when you have the OpenBSD bhyve zone in a shut-off state.

# zonecfg -z openbsd

remove device match=/dev/lofi/1
remove attr name=disk
verify
commit
exit

Unmount the lofi block device:

lofiadm -d /dev/lofi/1

Connecting to the installed OpenBSD bhyve zone

Console and optionally ssh should be available to connect to the zones:

# zlogin -C openbsd

or ssh into the OpenBSD bhyve zone if you have configured sshd .

ssh 192.168.1.67

Wrapping up

There are quite a few ways to configure zones, this has been one of them. Some other tips that you may find easier are as follows:

• In zonecfg , you can use zone templates to reduce typing a bit, so create -t bhyve in this case. Then you don’t need to set brand , or set ip-type .
• In the net section, you can set global-nic which makes the vnic get created and deleted automatically with VM boot.
• There is also the options to use zadm or zcage in assisting you with the setup of Zones on OmnioOS .

Thanks to Andy at citrus-it for these tips.