In 2014, I waxed lyrical about Arch Linux. At that time, I was running it on a virtual machine (surreal) and on two physical machines (jaenelle, menolly). Since then, I’ve acquired some more systems on which I run Arch: lisbon, my new daily-driver laptop, replacing jaenelle; and santiago, a media-player PC, which fulfils a role alyzon used to. I’ve also built some Arch User Repository packages, and managed a small repository of packages; and it’s largely through the latter that I’ve found myself increasingly frustrated by Arch’s packaging practices, and increasingly frustrated by running Arch itself.

To that end, I’ve been poking around distributions, trying to find something that fits my wants:

  1. When I want to get work done, I want to be able to get work done. I’m happy to unpick my own messes — hell, I generate enough of my own — but unpicking others’ sometimes frustrates me.

  2. Linux-based (and, consequently, GNU-based).

    This surprises a lot of people: surely I’d be running FreeBSD, right? Unfortunately, the hardware support story isn’t quite up to it on lisbon yet: past experience suggests suspend-resume will be too shaky. I also don’t want to run seL4 on bare metal, nor do I want to run the Hurd on a production system.

    The choice of Linux implies GNU, which I’d consider to be the de facto userland for a system running Linux.

  3. An upstream that both actively updates packages, and who gives some attention to quality assurance. Sadly, this trims lots of smaller one-person-shops which might be doing something interesting or innovative; but see (1).

  4. A coherent approach to distributing binary packages (or enough of one that I can grit my teeth). I’d much prefer to be able to snarf pre-built packages for the average case, and only in extremis have to build my own; I don’t want to rebuild the universe from scratch, even though it might look fun to do so.

  5. A friendly package manager (or enough of one that I can grit my teeth). I consider FreeBSD’s ports and packages systems remarkably painless to use, despite the occasional quirk. Arch’s pacman is pleasing, and I’ve never had any problems with it — though the invocation is somewhat arcane.

  6. Easy to provide patches upstream. Whilst I’ve used Arch for years, I’m still not quite sure how to patch others’ AUR packages except by maintaining my own patches. (Indeed, I eventually gave up on Git submodules and use Git subtrees with my patches atop them.)

  7. If I need to build packages, that build should be isolated from my live system. It should never be the case that dependencies were forgotten because nobody bothered to test in a clean system. Too many AUR packages fall into this category — Arch users building their own packages, you really should be using makechrootpkg! Going all the way to fully-reproducible builds certainly demonstrates the level of isolation I want, and it would be extremely cool to have, though isn’t necessarily what I really need.

  8. No reliance on a specific filesystem. I don’t run btrfs. I don’t want to run btrfs. (I run ZFS.)

  9. Don’t break things that work purely for the sake of new shiny things, until you can demonstrate that those shiny new things can meet or exceed what they set out to replace.

    X11 works perfectly fine! I’m probably one of the last people on the planet who uses XDMCP (on menolly, which is too feeble to manage much more). That, for example, as well as other things in my environment means switching to its notional replacement would be a long, tedious process of wheel-reinventing.

    Whilst my first encounters with it were troubling, PulseAudio now works perfectly fine — I have working audio on Linux! — and all I’ve seen of its notional replacement (via my partner, who runs Ubuntu) is inexplicable breakage.

On the whole, this leaves me with remarkably few options.

In spite of my occasional threats to build a distribution to my liking —

... at what point do I just roll my own distro?

— Jashank Jeremy (@jashankj) 2022-03-17T06:09:15.323Z

— I’d much prefer not to; though, unfortunately, I suspect I’m now well-equipped to do so. I would much prefer to delegate the dull stuff: I don’t really want to think about, say, the effort of maintaining package building services, or working out how to do quality-assurance.

On that point: it’s surprising to me that so few distros care about QA. Certainly, it’s an area I’ve found Arch lacking: as a ‘rolling-release’ distribution, it should be the case that, at any time, the set of packages available in repositories is self-consistent; yet I’ve run into serious issues in this regard. (Perhaps I’m the only person who tries to use Arch’s distributed packages for OCaml, though I seriously doubt this.)

I note with interest that the OCaml packages on Arch are broken; but, fascinatingly, nobody's noticed. Is there some minimum bar to packages being shipped that somehow isn't being met?

— Jashank Jeremy (@jashankj) 2022-03-17T06:09:14.658Z

Point 9 may be surprising; but consider: I’ve run a rolling-release distro with a great degree of flexibility in how I can manage and maintain the system. I can usually opt out of serious breaking changes. The last really seriously breaking change to Arch (to wit: systemd) seems to have occurred before I started running it.

So… where next?

Despite Curtis’ warnings, my first thought was Debian — probably Debian ‘testing’ — and I’d grit my teeth to the pain of dealing with the tower of horrors it uses for package management (dpkg and apt, urk).

... screw it, despite my vicious distaste for Debian-derived distros, I might try running rodete.

— Jashank Jeremy (@jashankj) 2022-03-17T11:36:05.981Z

And now, having spent about a month using it, I’m not happy with that decision.

I’m not a fan of the way packages are arranged; for example, I’ve had to hack around pain in: acpilight (missing!), docutils, libgccjit, xscreensaver, zsh plugins.

Nor am I a fan of AppArmor: too much time on Red Hat distros means I’ve got a feel for how to deal with SELinux.

Here’s a fun one: my network configuration, which worked perfectly fine on Arch, cannot bring up IPv6 by itself on Debian. I don’t know why, and despite having been digging around this issue for a few weeks (and with assistance from others), I am unable to find anything likely.

okay i'm stumped. why is my Debian installation ignoring IPv6?

— Jashank Jeremy (@jashankj) 2022-03-24T01:19:56.146Z

The packaging process is stuck, in places — e.g., debian:llvm-toolchain-14 cannot transition into testing due to a blocker bug that nobody seems to be investigating. I gave up and wound up using the upstream packages.

And then there’s packages that are just out-of-date — e.g., debian:dunst hasn’t been touched since October 2020.

Building Debian packages is complicated and alien. I appreciate that varying sizes of packaging rafts must be built atop an ocean of miscellaneous code — but dragging around the entire codebase as well seems weird. In the process of trying to update the Dunst package to the current upstream and contribute those changes back, I found much more pain than I’d’ve expected, and I still haven’t pushed my changes back upstream.

Whilst I appreciate the amount of effort expended in the bureaucracy of Debian’s policy requirements — having a well-documented and clear policy is superb! — I can’t help but feel it’s too much, and the returns are diminishing.

Debian’s downstream distros don’t bring joy, either. Having spent time debugging ZFS on Ubuntu —

ah yes, Ubuntu, Swahili for "we start with an out of date distro, then slap random technologies onto it that we don't understand and don't test". https://twitter.com/kuschku/status/1450247425852125185

— Jashank Jeremy (@jashankj) 2022-03-24T01:19:56.146Z

— I’m unimpressed and dissatisfied with how they did it. The prospect of a rolling-release Ubuntu fills me with dread. And, unfortunately, it’s downhill from there in DEB-space.

All up, the number of weird quirks and anachronisms makes Debian feel like an odd environment, too much so for my taste, and moreso than I was expecting. Curtis predicted this, and was entirely correct.

Well… where next, then?

Curtis’ initial suggestion was that I’d probably want a Red Hat-derived distro; after all, that’s what I recommended to him when he wanted to switch from Linux Mint, and, as far as I know, he’s been running either Fedora or CentOS on his machines since.

That seems like a good next step, so let’s see what exists in that space. Oddly, there don’t seem to be many rolling RPM-derived distros!

One option is CentOS Stream, which looks very plausible to me, though the spectrum of versions possible when describing oneself as “a midstream between Fedora Linux and RHEL” seems much wider than I’d like, and has the possibility of many more old packages than I’d be comfortable with.

Another option is OpenSuSE, who, since I last looked at their offering, have split their distribution into both a traditional point-wise release stream, ‘Leap’, and a rolling-release stream, ‘Tumbleweed’ — might I say, I adore these names. Certainly, Tumbleweed looks quite interesting, and it’s probably the next^2 place I’ll look.

But, at the moment, I’m most tempted to run Fedora ‘rawhide’, and pretend it’s just a normal rolling-release distro. Is this an utterly ridiculous idea? Probably! Am I going to cause myself lots of pain? Almost certainly!

However, Red Hat-derived distributions are probably the first Linux distributions I used, and certainly would be, after Arch, the ones I’ve used the most, so returning to them seems like a good choice, and Fedora has always struck me as a reasonably up-to-date distribution, even after release cuts, and even in spite of shipping not-quite-perfect products.

I’ll have probably more thoughts about this once I’ve spent some time using it: I suspect I’ll have enough thoughts to reëvaluate after another month or so of daily-driving.

Oh, and an addendum: how do I make all this work? ZFS, of course! On lisbon (and jaenelle), nearly all of the disk is one great big zpool, in which I have a half-dozen-odd boot environments. These tend to be entire distributions: so, for instance, lisbon/ROOT/arch has my hitherto-default Arch installation.

To install the systems takes a bit of thought, and I’ll write up separately my process of bootstrapping Debian and Fedora from scratch.