I Bootstrapped Fedora 'rawhide'

(Fedora devs who know what they’re doing: there’s probably nothing hugely amazing here, and instead just lots of horrors!)

After switching to Debian for a bit (and bootstrapping it from scratch!) I opted to switch to Fedora ‘rawhide’, and did so broadly non-destructively to my original Arch and Debian environment. How did I do it?

Short answer: ZFS boot environments. (And pain, and SQLite3.)

Long answer:

Create a boot environment.

This looks almost exactly the same as the way I did this for Debian.

Install a release identity.

For starters, I needed to install rpm on my host. (I suspect this probably could be worked around.) Like most distros that package cross-distro package managers, Debian have gone out of their way to make sure you don’t break your real system. Once you’ve done that:

rpm --root /lisbon/fedora --dbpath /var/lib/rpm --rebuilddb
alias rpm='rpm --force-debian --root /lisbon/fedora --dbpath /var/lib/rpm'

Next, you need to work out what packages to install.

The first bit is easy: you need to tell the nascent system that it’ll be a Fedora system, which you do by stuffing in the release packages, and the things that depends upon. Because Fedora 35 is the current stable, I’ll start with exactly it, as released.

These paths are all relative to /fedora/linux/releases/35/Everything/x86_64/os/ on whatever nearby mirror floats your boat and I dump all these RPMs into a directory stage0.

Packages/f/fedora-release-35-33.noarch.rpm
Packages/f/fedora-release-identity-basic-35-33.noarch.rpm
Packages/f/fedora-release-common-35-33.noarch.rpm
Packages/f/fedora-repos-35-1.noarch.rpm
Packages/f/fedora-repos-rawhide-35-1.noarch.rpm
Packages/f/fedora-gpg-keys-35-1.noarch.rpm

And that’s super easy to install.

rpm --install --verbose --hash stage0/*.rpm

Reinvent a dnf-based bootstrap.

This was probably unnecessary. It took me ages to discover microdnf, and I suspect that might be a better solution.

Now for the hard bit: because dnf is written in Python, it’s not as simple as grabbing a static binary and running it. Instead, you get to walk the dependency tree yourself! (I spent several minutes swearing after discovering microdnf.)

After some time, I eventually wound up with the following (where each indentation step is a new level of dependency resolution.)

Packages/d/dnf-4.9.0-1.fc35.noarch.rpm
 Packages/b/bash-5.1.8-2.fc35.x86_64.rpm
 Packages/p/python3-dnf-4.9.0-1.fc35.noarch.rpm
  Packages/d/dnf-data-4.9.0-1.fc35.noarch.rpm
  Packages/f/filesystem-3.14-7.fc35.x86_64.rpm
  Packages/g/glibc-2.34-7.fc35.x86_64.rpm
  Packages/l/libmodulemd-2.13.0-3.fc35.x86_64.rpm
  Packages/n/ncurses-libs-6.2-8.20210508.fc35.x86_64.rpm
  Packages/p/python3-3.10.0-1.fc35.x86_64.rpm
  Packages/p/python3-gpg-1.15.1-4.fc35.x86_64.rpm
  Packages/p/python3-hawkey-0.64.0-1.fc35.x86_64.rpm
  Packages/p/python3-libcomps-0.1.18-1.fc35.x86_64.rpm
  Packages/p/python3-libdnf-0.64.0-1.fc35.x86_64.rpm
  Packages/p/python3-rpm-4.17.0-1.fc35.x86_64.rpm
   Packages/a/audit-libs-3.0.6-1.fc35.x86_64.rpm
   Packages/b/basesystem-11-12.fc35.noarch.rpm
   Packages/b/bzip2-libs-1.0.8-9.fc35.x86_64.rpm
   Packages/e/elfutils-libelf-0.185-5.fc35.x86_64.rpm
   Packages/e/elfutils-libs-0.185-5.fc35.x86_64.rpm
   Packages/f/file-libs-5.40-9.fc35.x86_64.rpm
   Packages/g/glib2-2.70.0-5.fc35.x86_64.rpm
   Packages/g/glibc-common-2.34-7.fc35.x86_64.rpm
   Packages/g/glibc-langpack-en-2.34-7.fc35.x86_64.rpm
   Packages/g/gpgme-1.15.1-4.fc35.x86_64.rpm
   Packages/i/ima-evm-utils-1.3.2-3.fc35.x86_64.rpm
   Packages/l/libacl-2.3.1-2.fc35.x86_64.rpm
   Packages/l/libcap-2.48-3.fc35.x86_64.rpm
   Packages/l/libcomps-0.1.18-1.fc35.x86_64.rpm
   Packages/l/libdnf-0.64.0-1.fc35.x86_64.rpm
   Packages/l/libfsverity-1.4-4.fc35.x86_64.rpm
   Packages/l/libgcc-11.2.1-1.fc35.x86_64.rpm
   Packages/l/libreport-filesystem-2.15.2-6.fc35.noarch.rpm
   Packages/l/libsmartcols-2.37.2-1.fc35.x86_64.rpm
   Packages/l/libsolv-0.7.19-3.fc35.x86_64.rpm
   Packages/l/libstdc++-11.2.1-1.fc35.x86_64.rpm
   Packages/l/libyaml-0.2.5-6.fc35.x86_64.rpm
   Packages/l/libzstd-1.5.0-2.fc35.x86_64.rpm
   Packages/l/lua-libs-5.4.3-2.fc35.x86_64.rpm
   Packages/n/ncurses-base-6.2-8.20210508.fc35.noarch.rpm
   Packages/o/openssl-libs-1.1.1l-2.fc35.x86_64.rpm
   Packages/p/popt-1.18-6.fc35.x86_64.rpm
   Packages/p/python3-libs-3.10.0-1.fc35.x86_64.rpm
   Packages/r/rpm-build-libs-4.17.0-1.fc35.x86_64.rpm
   Packages/r/rpm-libs-4.17.0-1.fc35.x86_64.rpm
   Packages/r/rpm-sign-libs-4.17.0-1.fc35.x86_64.rpm
   Packages/s/setup-2.13.9.1-2.fc35.noarch.rpm
   Packages/s/sqlite-libs-3.36.0-3.fc35.x86_64.rpm
   Packages/x/xz-libs-5.2.5-7.fc35.x86_64.rpm
   Packages/z/zlib-1.2.11-30.fc35.x86_64.rpm
    Packages/c/ca-certificates-2021.2.50-3.fc35.noarch.rpm
    Packages/c/crypto-policies-20210819-1.gitd0fdcfb.fc35.noarch.rpm
    Packages/e/elfutils-default-yama-scope-0.185-5.fc35.noarch.rpm
    Packages/e/expat-2.4.1-2.fc35.x86_64.rpm
    Packages/g/gdbm-libs-1.20-2.fc35.x86_64.rpm
    Packages/g/gnupg2-2.3.2-2.fc35.x86_64.rpm
    Packages/g/gnutls-3.7.2-2.fc35.x86_64.rpm
    Packages/j/json-c-0.15-2.fc35.x86_64.rpm
    Packages/k/keyutils-libs-1.6.1-3.fc35.x86_64.rpm
    Packages/l/libassuan-2.5.5-3.fc35.x86_64.rpm
    Packages/l/libattr-2.5.1-3.fc35.x86_64.rpm
    Packages/l/libcap-ng-0.8.2-6.fc35.x86_64.rpm
    Packages/l/libffi-3.1-29.fc35.x86_64.rpm
    Packages/l/libgomp-11.2.1-1.fc35.x86_64.rpm
    Packages/l/libgpg-error-1.42-3.fc35.x86_64.rpm
    Packages/l/libmount-2.37.2-1.fc35.x86_64.rpm
    Packages/l/libnsl2-1.3.0-4.fc35.x86_64.rpm
    Packages/l/librepo-1.14.2-1.fc35.x86_64.rpm
    Packages/l/libselinux-3.2-4.fc35.x86_64.rpm
    Packages/l/libtirpc-1.3.2-1.fc35.x86_64.rpm
    Packages/l/libuuid-2.37.2-1.fc35.x86_64.rpm
    Packages/l/libxcrypt-4.4.26-4.fc35.x86_64.rpm
    Packages/l/libxml2-2.9.12-6.fc35.x86_64.rpm
    Packages/m/mpdecimal-2.5.1-2.fc35.x86_64.rpm
    Packages/p/pcre-8.45-1.fc35.x86_64.rpm
    Packages/p/python-pip-wheel-21.2.3-2.fc35.noarch.rpm
    Packages/p/python-setuptools-wheel-57.4.0-1.fc35.noarch.rpm
    Packages/r/readline-8.1-3.fc35.x86_64.rpm
    Packages/r/rpm-4.17.0-1.fc35.x86_64.rpm
    Packages/t/tpm2-tss-3.1.0-3.fc35.x86_64.rpm
    Packages/t/tzdata-2021b-1.fc35.noarch.rpm
    Packages/z/zchunk-libs-1.1.15-2.fc35.x86_64.rpm
     Packages/c/coreutils-8.32-31.fc35.x86_64.rpm
     Packages/c/curl-7.78.0-3.fc35.x86_64.rpm
     Packages/g/gmp-6.2.0-7.fc35.x86_64.rpm
     Packages/g/grep-3.6-4.fc35.x86_64.rpm
     Packages/k/krb5-libs-1.19.2-2.fc35.x86_64.rpm
     Packages/l/libarchive-3.5.2-2.fc35.x86_64.rpm
     Packages/l/libblkid-2.37.2-1.fc35.x86_64.rpm
     Packages/l/libcom_err-1.46.3-1.fc35.x86_64.rpm
     Packages/l/libcurl-7.78.0-3.fc35.x86_64.rpm
     Packages/l/libgcrypt-1.9.4-1.fc35.x86_64.rpm
     Packages/l/libidn2-2.3.2-3.fc35.x86_64.rpm
     Packages/l/libksba-1.6.0-2.fc35.x86_64.rpm
     Packages/l/libsepol-3.2-3.fc35.x86_64.rpm
     Packages/l/libtasn1-4.16.0-6.fc35.x86_64.rpm
     Packages/l/libunistring-0.9.10-14.fc35.x86_64.rpm
     Packages/n/nettle-3.7.3-2.fc35.x86_64.rpm
     Packages/n/npth-1.6-7.fc35.x86_64.rpm
     Packages/o/openldap-2.4.59-3.fc35.x86_64.rpm
     Packages/p/p11-kit-0.23.22-4.fc35.x86_64.rpm
     Packages/p/p11-kit-trust-0.23.22-4.fc35.x86_64.rpm
     Packages/p/pcre2-10.37-4.fc35.x86_64.rpm
     Packages/s/sed-4.8-8.fc35.x86_64.rpm
     Packages/s/shadow-utils-4.9-3.fc35.x86_64.rpm
      Packages/a/alternatives-1.19-1.fc35.x86_64.rpm
      Packages/c/coreutils-common-8.32-31.fc35.x86_64.rpm
      Packages/c/cyrus-sasl-lib-2.1.27-13.fc35.x86_64.rpm
      Packages/g/gawk-5.1.0-4.fc35.x86_64.rpm
      Packages/l/libbrotli-1.0.9-6.fc35.x86_64.rpm
      Packages/l/libnghttp2-1.45.1-1.fc35.x86_64.rpm
      Packages/l/libpsl-0.21.1-4.fc35.x86_64.rpm
      Packages/l/libsemanage-3.2-4.fc35.x86_64.rpm
      Packages/l/libsigsegv-2.13-3.fc35.x86_64.rpm
      Packages/l/libssh-0.9.6-1.fc35.x86_64.rpm
      Packages/l/libverto-0.3.2-2.fc35.x86_64.rpm
      Packages/l/lz4-libs-1.9.3-3.fc35.x86_64.rpm
      Packages/p/pcre2-syntax-10.37-4.fc35.noarch.rpm
       Packages/l/libssh-config-0.9.6-1.fc35.noarch.rpm
       Packages/m/mpfr-4.1.0-8.fc35.x86_64.rpm
       Packages/p/publicsuffix-list-dafsa-20210518-2.fc35.noarch.rpm

I started doing this by hand, and eventually wrote some scripts, because it started getting tricky — one useful (to me) fact is that the repository metadata is all usable via a SQLite database, so I downloaded that, and hacked together:

#!/usr/bin/env zsh

primary=core/9103df16446e4e0b7eb02fe94936b7ec20c2caac436acedf9a8ba19f52c5af51-primary.sqlite

QUERY()   { sqlite3 -readonly $primary "select packages.location_href from packages $@" }
name()    { QUERY "where packages.name like '$@';" }
provide() { QUERY "join provides on (packages.pkgKey = provides.pkgKey) where provides.name like '$@';" }

op="$1"; shift
case "$op" in
	(name)
		name "$@"
		;;
	(provide)
		provide "$@"
		;;
	(*)
		exit 1
		;;
esac

# >>> ./match name libssh
# Packages/l/libssh-0.9.6-1.fc35.i686.rpm
# Packages/l/libssh-0.9.6-1.fc35.x86_64.rpm

With that package list in hand, ,and then downloaded into stage1:

rpm --install --verbose --hash stage1/*.rpm

It turns out this package set is a non-trivial portion of a basic Fedora system: remarkably little is needed afterwards to be able to step through more of the installation process.

Finish the bootstrap of Fedora 35.

The next thing to do is to step into our new system, and start the process of lifting it from Fedora 35 to Fedora rawhide — presently, rawhide is Fedora 37 — which is surprisingly painless.

systemd-nspawn --quiet --directory=/lisbon/fedora --register=yes --as-pid2

dnf shell
install dnf-command(system-upgrade)
update
run

There’s two important things happening here: first, we update to the newest packages, which is absolutely a prerequisite to any sort of release-hopping operations; and second, we summon up the release-hopping tool. It’s important to note here that I’ve not installed anything beyond the bootstrapping package set: I’m going to install more only once I’ve arrived on rawhide. I use dnf shell to run the whole thing in one transaction.

Next, we need to do some weird key juggling. I’m not quite sure why this isn’t running automatically.

rpm -K /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-3[567]-x86_64

Upgrade to Fedora rawhide.

This is the fun bit: download and install the new system!

dnf system-upgrade download --refresh --releasever=rawhide
export DNF_SYSTEM_UPGRADE_NO_REBOOT=True
dnf system-upgrade reboot --releasever=rawhide
dnf system-upgrade upgrade --releasever=rawhide

Now, we “reboot”.

exit

systemd-nspawn --quiet --directory=/lisbon/fedora --register=yes --as-pid2

Set up additional repositories.

At the moment, the OpenZFS folks don’t have any Fedora 36 or later repositories. I’m therefore assuming that the Fedora 35 packages will work, and this so far has been true.

dnf install https://zfsonlinux.org/fedora/zfs-release.fc35.noarch.rpm
rpm -K /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

I’d also like RPM Fusion.

dnf install \
https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-rawhide.noarch.rpm \
https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-rawhide.noarch.rpm
rpm -K /etc/pki/rpm-gpg/RPM-GPG-KEY-rpmfusion-{,non}free-fedora-rawhide

For completeness, I’m doing the key juggling myself again. These do, it turns out, happen automatically.

Install ZFS bits.

We need to install everything except ZFS here —

dnf shell
install @core
install grub2-efi-x64 grub2-efi-x64-modules
install grub2-common grub2-tools grub2-tools-efi
install shim-x64
install efibootmgr
install kernel kernel-headers kernel-devel
install python3-dnf-plugin-post-transaction-actions
update
run

— because the next thing I do is this extremely evil, cursed, horrible thing that you should absolutely not ever do at all under any circumstances:

dnf remove kernel{,-core,-devel,-modules}
dnf install \
--disablerepo=rawhide \
--enablerepo=fedora,updates --setopt=releasever=36 \
kernel{,-core,-devel,-modules}

Woah, why am I throwing away the rawhide kernel? Because, it turns out, ZFS collides with it: OpenZFS isn’t quite ready for the 5.18 kernels, and after some cajoling I found that the Fedora 36-Beta kernels work fine — why yes, I’m doing this during the pointy end of the Fedora 36 release cycle, so everything’s full of exciting weirdness! — so I’m opting into them over rawhide. I suspect I’ll continue to track released kernels, but run the bleeding-edge user-lands. (I sure wish the Modularity work was possible here!)

With all that dealt with, at last:

dnf install zfs zfs-dracut

Install the rest of the universe.

ccat ~/etc/packages.fedora.lisbon > /lisbon/fedora/var/tmp/pkgs

I also wound up doing some group marking too.

dnf install $(cat /var/tmp/pkgs)

Interlude: fix passwd, shadow, group, gshadow.

Here’s a horrible quirk to deal with: I can’t directly reuse the system credentials databases (i.e., passwd(5), shadow(5), group(5), gshadow(5)) between these separate system roots, because the base set of identities between the systems differs; and then the identities generated by systemd’s sysusers.d(5) mechanism tend to have no fixed UIDs or GIDs specified, so I now have three separate UID/GID tables.

I started trying to unify them by hand, spent some time tearing my hair out, and eventually reached the point of wanting to build a tool to do it. This caused more tearing of hair (and lots of grumbling at z3 — that should be the ideal tool for the job, except trying to express my problem to it proved to be beyond my capabilities).

I eventually hacked up generate-sys-cred-tables, which let me define users and groups using a single configuration file, and then generates a matching set of passwd, shadow, group, and gshadow files.

I then hacked up some tools to deal with rewriting UIDs and GIDs in a system: generate-uid-gid-deltas takes current and new credentials tables, and generates a mapping of old-to-new identifiers; and translate-mtree-uids-gids, takes mtree(8) on standard input, and applies the generated mappings from generate-uid-gid-deltas; and install-new-sys-cred-tables swaps from the current to the new tables within that particular sysroot.

(translate-mtree-uids-gids probably looks like it should be really slow — but because mtree(8) was more CPU-bound, the performance overhead here was effectively negligible.)

Now I change directories and run this mess. I previewed the changes, and then ran the rewrite.

generate-uid-gid-deltas /lisbon/fedora
cd /lisbon/fedora
sudo mtree -c | translate-uids-gids ~/ch.{passwd,shadow}.fedora | sudo mtree
                  [... lots of output elided; check it makes sense!]
                  [commit by making the final command `mtree -u']

And I install the tables using install-new-sys-cred-tables.

cd /lisbon/fedora/etc
sudo install-new-sys-cred-tables

(I also ran this in my Arch install. I’ll eventually run it in my Debian install, but I’m not quite mad enough to change the creds tables this substantially on a running *nix system…)

Do post-installation configuration.

This looks almost exactly the same as the way I did this for Debian.

One notable addition: sort out SELinux. I’m going to run permissive for a bit, until I get used to dealing with it again — whilst, yes, I do run SELinux in my current Debian environment, the default Fedora policy set is different, and definitely needs more consciousness.

sed -i -e 's/^SELINUX=.*/SELINUX=permissive/' etc/selinux/config
systemd-nspawn --quiet --directory=/lisbon/fedora --register=yes --as-pid2
              [force a full relabel.]
restorecon -T64 -p -F -r /

Beware, though, that doing a normal mass relabel will clobber any labels on the other system-roots. You’ll need to exercise some care about how to apply labels to things like home directories or any other content to be shared between systems. In particular, I had to do a bit of experimenting to work out how/when/what to relabel.

(There’s almost definitely stuff missing that I’ve forgotten entirely, and which I’m sure I’ll remember eventually.)

Enjoy.

This is, of course, the interesting bit.