Migrate TrueNAS Core to Sylve

For years I ran TrueNAS Core on my home fileserver. This was an awesome, easy way of running FreeBSD + ZFS + tools without a lot of manual maintenance. Unfortunately TrueNAS decided to start using Linux as the basis of TrueNAS and I love using FreeBSD. Fortunately a new project is on the horizon which looks promising:

 Sylve

 I'm eager to try out Sylve and will write down my plan and the results in this blog.

Some useful facts

  • TrueNAS Core installs full images containing the OS, middleware and UI. The last version was based on FreeBSD 13.3 and uses Boot Environments to manage versions.
  • Sylve wants FreeBSD 15.0 or later pre-installed and only installs middleware and UI on top of it.
  • I use my NAS mainly for Samba (Time Machine) and ZRepl (zfs+ssh send-receive backups).
  • The machine: dmesg.
  • root@freenas[~]# gpart show
    =>       40  234441568  ada0  GPT  (112G)
             40       1024     1  freebsd-boot  (512K)
           1064  234422272     2  freebsd-zfs  (112G)
      234423336      18272        - free -  (8.9M)

    =>        40  7814037088  ada2  GPT  (3.6T)
              40          88        - free -  (44K)
             128     4194304     1  freebsd-swap  (2.0G)
         4194432  7809842696     2  freebsd-zfs  (3.6T)

    =>        40  7814037088  ada3  GPT  (3.6T)
              40          88        - free -  (44K)
             128     4194304     1  freebsd-swap  (2.0G)
         4194432  7809842696     2  freebsd-zfs  (3.6T)

    =>       40  234441568  ada1  GPT  (112G)
             40       1024     1  freebsd-boot  (512K)
           1064  234422272     2  freebsd-zfs  (112G)
      234423336      18272        - free -  (8.9M)

  • root@freenas[~]# zpool status

      pool: freenas-boot

     NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            ada1p2    ONLINE       0     0     0
            ada0p2    ONLINE       0     0     0

      pool: zdata
    NAME                                                STATE     READ WRITE CKSUM
        zdata                                               ONLINE       0     0     0
          mirror-0                                          ONLINE       0     0     0
            gptid/ac5968d1-c9ff-11ea-9bab-002170466cda.eli  ONLINE       0     0     0
            gptid/ad3475fc-c9ff-11ea-9bab-002170466cda.eli  ONLINE       0     0     0 
  • root@freenas[~]# mount
    freenas-boot/ROOT/13.3-U1.2 on / (zfs, local, noatime, nfsv4acls)
    devfs on /dev (devfs)
    tmpfs on /etc (tmpfs, local)
    tmpfs on /mnt (tmpfs, local)
    tmpfs on /var (tmpfs, local)
    fdescfs on /dev/fd (fdescfs)
    zdata on /mnt/zdata (zfs, local, nfsv4acls)
    zdata/backups on /mnt/zdata/backup (zfs, local, read-only, nfsv4acls)
    zdata/.system on /var/db/system (zfs, local, nfsv4acls)
    zdata/.system/cores on /var/db/system/cores (zfs, local, nfsv4acls)
    zdata/.system/samba4 on /var/db/system/samba4 (zfs, local, nfsv4acls)
    zdata/.system/syslog-646f8dae97d646cc8946ddeb0ca79d97 on /var/db/system/syslog-646f8dae97d646cc8946ddeb0ca79d97 (zfs, local, nfsv4acls)
    zdata/.system/rrd-646f8dae97d646cc8946ddeb0ca79d97 on /var/db/system/rrd-646f8dae97d646cc8946ddeb0ca79d97 (zfs, local, nfsv4acls)
    zdata/.system/configs-646f8dae97d646cc8946ddeb0ca79d97 on /var/db/system/configs-646f8dae97d646cc8946ddeb0ca79d97 (zfs, local, nfsv4acls)
    zdata/.system/webui on /var/db/system/webui (zfs, local, nfsv4acls)
    zdata/.system/services on /var/db/system/services (zfs, local, nfsv4acls)
    fdescfs on /var/run/samba/fd (fdescfs) 
  • [root@freenas ~]# ifconfig 
    em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e524bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether xx:xx:xx
        inet6 fe80::xx:xx:xx%em0 prefixlen 64 scopeid 0x1
        inet6 xx:xx:xx prefixlen 64 autoconf
        inet xx:xx:xx netmask 0xffffff00 broadcast xx:xx:xx
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog

So it has 2 mirrored SSDs containing the OS and 2 mirrored HDDs containing my data.
The Boot Environment is one big filesystem containing the whole OS. Only /var/db/system is a filesystem on the HDD containing TrueNAS specific config files and statistics.

Personal requirement

  • Keep existing disks with all data untouched.
  • Keep existing Boot Environments so I can revert or lookup current configuration settings. 

The Plan

  1. Create new Boot Environment for plain FreeBSD 15.0.
  2. Install Sylve and recommended packages.
  3. Hope that it recognizes my existing pools.
  4. Configure Samba and ZFS send-receive.

The Execution

1. Create new Boot Environment for plain FreeBSD 15.0 

# cd /tmp
# fetch "https://download.freebsd.org/releases/amd64/15.0-RELEASE/kernel.txz"
# fetch "https://download.freebsd.org/releases/amd64/15.0-RELEASE/base.txz"
# beadm create 15.0-R
# mkdir /tmp/15.0
# beadm mount 15.0-R /tmp/15.0/
# cd /tmp/15.0/
# rm -rxf *
# chflags -R noschg *
# rm -rxf *
# tar xvf /tmp/base.txz
# tar xvf /tmp/kernel.txz
# vim etc/rc.conf
hostname="xxxxxx"

ifconfig_DEFAULT="SYNCDHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"

zfs_enable="YES"

sshd_enable="YES"
powerd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
# chroot /tmp/15.0 /bin/sh
# passwd
<set root password>
# adduser
<add a user to ssh into the server, make sure it is in group wheel to become root later>
# exit
# vim /boot/loader.conf
zfs_load="YES"
autoboot_delay="3"
# beadm unmount 15.0-R
# beadm activate 15.0-R
# shutdown -r now
<wait for reboot and log in with the added user (the ssh hostkey changed so I had to edit my .ssh/known_hosts file>
<celebrate>

2. Install Sylve and recommended packages 

My NAS is now running 15.0-RELEASE. Pool freenas-boot is found. The zdata pool needs to be unlocked with the geli keys.
Bectl list shows all previous TrueNAS Boot Environments.

Well, as we now have a working plain FreeBSD 15 server, from here it was pretty straightforward. Follow the Sylve guide: https://sylve.io/getting-started/.
NB: While writing this blog a port of Sylve became available: https://www.freshports.org/sysutils/sylve/, so you can do pkg install sylve.

Some custom steps I did:

2.1 Setup geli to make my data pool available 

All my Boot Environments are available. The tool to use is now bectl instead of beadm.

# bectl list -c creation
BE                             Active Mountpoint Space Created
default                        -      -          3.19M 2020-07-19 23:45
Initial-Install                -      -          2.89M 2020-07-19 23:48
11.3-U4                        -      -          1023M 2020-07-25 10:13
11.3-U4.1                      -      -          1023M 2020-08-11 17:05
FreeNAS-12.0-RC1               -      -          1.06G 2020-09-19 11:22
12.0-RELEASE                   -      -          1.06G 2020-10-20 23:02
12.0-U1                        -      -          1.15G 2020-12-10 10:14
[...]
13.3-U1.2                      -      -          1.24G 2025-05-08 10:22
15.0-R                         NR     /          47.1G 2026-03-29 17:27
 

# bectl mount 13.3-U1.2
# mkdir /etc/geli
# cp /tmp/be_mount.oxgP/data/geli/002a0060-5055-49d0-8488-631c83e73ca3.key /etc/geli/storage.key
Added some lines to /etc/rc.conf:
# TrueNAS said in /etc/rc.conf:
# Do not mark to autodetach otherwise ZFS gets very unhappy.
# XXX-TODO: is this still needed?
geli_autodetach="NO"
geli_groups="storage"
geli_storage_flags="-p -k /etc/geli/storage.key"
geli_storage_devices="ada2p2 ada3p2"

2.2 Setup my SSL/TLS certificate

Added a script that downloads certifcates from my Vault to /usr/local/etc/sylve/fullchain.pem.
Added tlsConfig to /usr/local/etc/sylve/config.json as described in the Sylve Guide: tlsConfig.

2.3 Setup gmirror for the swap

Added geom_mirror_load="YES" to /boot/loader.conf.
# gmirror label -F swap0 /dev/ada2p1 /dev/ada3p1
Added this line to /etc/fstab:
/dev/mirror/swap0.eli   none    swap    sw      0       0
 

3. Hope that it recognizes my existing pools

 All my disks and data are available.

4. Configure Samba and ZFS send-receive

Exercise for the reader. I'm still looking into what is possible.

Comments

Post a Comment

Popular posts from this blog

Waiting for clang. Forever and ever.

FreeBSD compiles with LLVM /clang by default. Part of this is compiling the compiler to accommodate upgrades of the compiler. On my laptop FreeBSD compiles in about 2 hours. But clang takes 2 hours on its own. The culprit is that now and than a small fix is made to the compiler which triggers 2 hours of extra building. It is also possible to install llvm as a pkg. This saves you hours of compiling. I tried to built base with the port, but it did not succeed yet. I now have setup FreeBSD in a container (FreeBSD jail) which helps greatly in fixing mistakes instead of a non bootable system without a working compiler I can now just reinstall the container from a working host. My changes to make this working: This is on: FreeBSD sjakie 13.0-CURRENT FreeBSD 13.0-CURRENT #11 r364932M: Sat Aug 29 03:21:18 CEST 2020 Add to /etc/src.conf : WITHOUT_CLANG=true WITHOUT_CLANG_IS_CC=true WITHOUT_CLANG_BOOTSTRAP=true WITHOUT_LLD=true WITHOUT_LLD_IS_LD=true WITHOUT_LLD_BOOTSTRAP=true WITHOUT_LLDB=true...
Read more

FreeBSD running vlan and bridge on one shared interface

 I have this setup on FreeBSD 13.2 on my Raspberry PI: vlans_ue0="vlan3" create_args_vlan3="vlan 3" cloned_interfaces="bridge0 bridge3" ifconfig_bridge0="addm ue0 SYNCDHCP" ifconfig_bridge0_ipv6="inet6 auto_linklocal accept_rtadv" ifconfig_ue0="up" ifconfig_bridge3="addm vlan3" ifconfig_vlan3="up" So: +--vlan3--bridge3--jailProxy | ue0-+ | +--bridge0--jailWifi Networking from the host works fine. Networking from jailWifi works fine. But... jailProxy can send DHCP to the network in vlan 3. But the reply goes to jailWifi. Does bridge0 have precedence over vlan3? Could I let vlan3 handle packages with the proper vlan tag only and let the rest of the traffic go to bridge0? Ok. While typing this I came up with a solution which I tested and works. ue0--bridge0-+--jailWifi | +--epair0a--epair0b--vlan3--bridge3--jailProxy So now I have this DMZ bridge3 on its own VLAN. NB: ...
Read more