Other FreeBSD Version in ZFS Boot Environment

The first FreeBSD 12.3-PRERELEASE snapshots are finally available. This means we can try them in a new ZFS Boot Environment without touching out currently running 13.0-RELEASE system. We can not take the usual path with creating new BE from our current one and upgrade it to newer version because 12.3 has older major version then the 13.0 one.

This is kinda a paradox in the FreeBSD release process that when released the 12.3-RELEASE will have some newer commits and features then older 13.0-RELEASE which was released earlier this year. Of course not all things that have been committed to HEAD goes into 12-STABLE or 13-STABLE automatically – but most of them do. Only the biggest changes will be limited only to 14.0-RELEASE – of course probably somewhere in the middle of 2022 when it will be having its release process.

One note about ZFS filesystem on FreeBSD. People often confuse ‘real’ ZFS Boot Environments with its trying-to-be substitutes like BTRFS snapshots or snapshots used by Ubuntu with zsysctl(8) command. Unfortunately they are only snapshots and are not full writable clones (or entire separate ZFS datasets). They can freeze your system in time so you will be able to get back to working configuration after updating packages for example – but You will not be able to install other separate version of a system as other ZFS dataset making it another independent ZFS Boot Environment.

Create New ZFS Dataset

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01

host # zfs list -r zroot/ROOT
NAME                        USED  AVAIL     REFER  MOUNTPOINT
zroot/ROOT                 12.8G  96.8G       88K  none
zroot/ROOT/13.0.w520       12.8G  96.8G     11.6G  /
zroot/ROOT/13.0.w520.safe     8K  96.8G     11.1G  /

host # zfs create -o mountpoint=/ -o canmount=off zroot/ROOT/12.3

host # beadm list
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           12.8G 2021-09-14 17:27
13.0.w520.safe -      -            1.2G 2021-10-18 10:01
12.3           -      -           96.0K 2021-10-18 13:14

Install FreeBSD 12.3-PRERELEASE

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # beadm list
BE             Active Mountpoint     Space Created
13.0.w520      NR     /              12.8G 2021-09-14 17:27
13.0.w520.safe -      -               1.2G 2021-10-18 10:01
12.3           -      /var/tmp/12.3  96.0K 2021-10-18 13:14

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/base.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  173M  100  173M    0     0  1889k      0  0:01:33  0:01:33 --:--:-- 2228k

host # exa -1 /var/tmp/12.3
bin
boot
dev
etc
lib
libexec
media
mnt
net
proc
rescue
root
sbin
tmp
usr
var
COPYRIGHT
sys

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/kernel.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 43.3M  100 43.3M    0     0  1733k      0  0:00:25  0:00:25 --:--:-- 1663k

host # exa -lh /var/tmp/12.3/boot/kernel/kernel
Permissions Size User Date Modified    Name
.r-xr-xr-x   37M root 2021-10-14 06:31 /var/tmp/12.3/boot/kernel/kernel

host # curl -o - https://download.freebsd.org/ftp/snapshots/amd64/12.3-PRERELEASE/lib32.txz \
         | tar --unlink -xpf - -C /var/tmp/12.3

host # exa -ld /var/tmp/12.3/usr/lib32
drwxr-xr-x - root 2021-10-18 13:45 /var/tmp/12.3/usr/lib32

Install Same Packages as on Host

With the pkg prime-list we will get all installed by hand pkg(8)packages from our currently running system. You may omit this section or just install packages that you need instead all of them.

host # pkg prime-list > /var/tmp/12.3/pkg.prime-list

host # chroot /var/tmp/12.3 /bin/sh

(BE) # export PS1="BE # "

BE # mount -t devfs devfs /dev

BE # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf

BE # pkg install -y $( cat pkg.prime-list )
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/latest, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
Installing pkg-1.17.2...
Extracting pkg-1.17.2: 100%
Updating FreeBSD repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%    6 MiB   1.3MB/s    00:05
Processing entries: 100%
FreeBSD repository update completed. 31294 packages processed.
All repositories are up to date.
Updating database digests format: 100%
pkg: No packages available to install matching 'chromium' have been found in the repositories
pkg: No packages available to install matching 'drm-fbsd13-kmod' have been found in the repositories
pkg: No packages available to install matching 'geany-gtk2' have been found in the repositories
pkg: No packages available to install matching 'ramspeed' have been found in the repositories
pkg: No packages available to install matching 'vim-console' have been found in the repositories

As we can see some of the packages that we have installed in the FreeBSD 13.0-RELEASE system are not currently available in the ‘latest‘ pkg(8) branch for the FreeBSD 12.3-PRERELEASE system. This sometimes happens when the build of such package will fail – but you may assume that such package will be available in a week or so as that is the period in which pkg(8) packages are (re)build in the ‘latest‘ branch.

We will now remove the missed packages and also rename some packages that may have different names for 12.x version of FreeBSD.

BE # sed -i '' \
         -e s/drm-fbsd13-kmod/drm-kmod/g \
         -e s/geany-gtk2/geany/g \
         -e s/vim-console/vim-tiny/g \
         pkg.prime-list

BE # pkg install -y $( cat pkg.prime-list | grep -v -e chromium -e ramspeed )
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1072 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        (...)

Number of packages to be installed: 1072

The process will require 11 GiB more space.
2 GiB to be downloaded.
(...)

BE # rm pkg.prime-list

After hour or so later our packages have been installed.

BE # pkg stats
Local package database:
        Installed packages: 1073
        Disk space occupied: 11 GiB

Remote package database(s):
        Number of repositories: 1
        Packages available: 31294
        Unique packages: 31294
        Total size of packages: 96 GiB

Copy Configuration Files

You can now reboot to plain and unconfigured FreeBSD system but you may as well copy your configuration files from your current working installation. These are the files I have copied.

First files from the Base System /etc and /boot places.

host # for I in /boot/loader.conf \
                /etc/hosts \
                /etc/fstab \
                /etc/rc.conf \
                /etc/sysctl.conf \
                /etc/wpa_supplicant.conf \
                /etc/jail.conf \
                /etc/devfs.rules \
                /etc/resolv.conf
       do
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"
       done
/boot/loader.conf
/etc/hosts
/etc/fstab
/etc/rc.conf
/etc/sysctl.conf
/etc/wpa_supplicant.conf
/etc/jail.conf
/etc/devfs.rules
/etc/resolv.conf

Now the files for installed packages under /usr/local/etc dir.

host # for I in /usr/local/etc/X11/xorg.conf.d/* \
                /usr/local/etc/X11/xdm/{Xresources,Xsetup_0} \
                /usr/local/etc/automount.conf \
                /usr/local/etc/sudoers \
                /usr/local/etc/doas.conf \
                /usr/local/etc/zshrc
       do
         cp "${I}" /var/tmp/12.3/"${I}"
         echo "${I}"
       done
/usr/local/etc/X11/xorg.conf.d/card.conf
/usr/local/etc/X11/xorg.conf.d/flags.conf
/usr/local/etc/X11/xorg.conf.d/keyboard.conf
/usr/local/etc/X11/xorg.conf.d/touchpad.conf
/usr/local/etc/X11/xdm/Xresources
/usr/local/etc/X11/xdm/Xsetup_0
/usr/local/etc/automount.conf
/usr/local/etc/sudoers
/usr/local/etc/doas.conf
/usr/local/etc/zshrc

Add Users and Set Passwords

You should now add your regular user and set passwords for both your user and root account.

BE # pw useradd vermaden -u 1000 -d /home/vermaden -G wheel,operator,video,network,webcamd,vboxusers

BE # passwd root

BE # passwd vermaden

Reboot Into New ZFS Boot Environment

You may now exit the chroot(8) of that ZFS Boot Environment and reboot. In the FreeBSD loader(8) menu select the 12.3 boot environment.

BE # exit

host # umount /var/tmp/12.3/dev

host # beadm unmount 12.3
Unmounted successfully

host # beadm list -D
BE             Active Mountpoint  Space Created
13.0.w520      NR     /           11.3G 2021-09-14 17:27
13.0.w520.safe -      -           11.1G 2021-10-18 10:01
12.3        -      -            9.5G 2021-10-18 13:14

host # shutdown -r now

Testing New System

The 12.3-PRERELEASE system started fine for me. I was able to login and use system as usual. One important thing to note … the ZFS pools. I have another newer ZFS pool with zstd compression enabled … and I was not able to import that ZFS pool as FreeBSD 12.3-PREELEASE does not use OpenZFS 2.0 but an older FreeBSD in-house ZFS version. Keep that in mind … but you can also install newer OpenZFS from the FreeBSD Ports and this is what we will now do.

# pkg install -y openzfs openzfs-kmod
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        openzfs: 2021090800
        openzfs-kmod: 2021090800

Number of packages to be installed: 2

The process will require 22 MiB more space.
4 MiB to be downloaded.
[1/2] Fetching openzfs-2021090800.pkg: 100%    3 MiB 975.3kB/s    00:03
[2/2] Fetching openzfs-kmod-2021090800.pkg: 100%    1 MiB 591.2kB/s    00:02
Checking integrity... done (0 conflicting)
[1/2] Installing openzfs-kmod-2021090800...
[1/2] Extracting openzfs-kmod-2021090800: 100%
pkg: Cannot open /dev/null:No such file or directory
[2/2] Installing openzfs-2021090800...
[2/2] Extracting openzfs-2021090800: 100%
=====
Message from openzfs-kmod-2021090800:

--
Amend /boot/loader.conf as follows to use this module:

- change zfs_load="YES" to NO
- change opensolaris_load="YES" to NO
- add openzfs_load="YES"
- (for ARM64) add cryptodev_load="YES"
=====
Message from openzfs-2021090800:

--
Ensure that any zfs-related commands, such as zpool, zfs, as used in scripts
and in your terminal sessions, use the correct path of /usr/local/sbin/ and
not the /sbin/ commands provided by the FreeBSD base system.

Consider setting this in your shell profile defaults!

We will now have to modify our /boot/loader.conf file.

host # beadm mount 12.3 /var/tmp/12.3
Mounted successfully on '/var/tmp/12.3'

host # chroot /var/tmp/12.3

BE # cp /boot/loader.conf /boot/loader.conf.ZFS

BE # vi /boot/loader.conf

BE # diff -u /boot/loader.conf.ZFS /boot/loader.conf
--- /boot/loader.conf.ZFS       2021-10-19 10:57:04.180732000 +0000
+++ /boot/loader.conf   2021-10-19 10:57:23.992145000 +0000
@@ -12,7 +12,8 @@

 # MODULES - BOOT
   geom_eli_load=YES
-  zfs_load=YES
+  zfs_load=NO
+  openzfs_load=YES

 # DISABLE /dev/diskid/* ENTRIES FOR DISKS
   kern.geom.label.disk_ident.enable=0

BE # shutdown -r now

After reboot and trying again I was able to import that newer ZFS pool.

Hope that you will find that guide useful.

Feel free to add your suggestions.