Packaging Hell Problems
by Mithrandir
Because the move to a new blog was imminent, I decided not to post anymore on this blog. However, I’ve encountered a problem which prompted me to do this.
I was using Arch since the October of the last year. I was a proud user, it was a system in which my productivity was high. I ignored all problems reported by others who were bitten by the unsafe updates. Until it happened to me.
The trickster update required making /lib a symlink to /usr/lib, something which no other operating system that I know has done before. “Maybe they had their reasons“, I thought and decided to update — aided in this decision by the fact that a new nvidia upgrade was on the way and I needed it desperately for suspend reasons.
Wrong move. The update couldn’t be done without human intervention so I opened the wiki and followed the instructions there. Removed /lib instead of removing /lib/modules — though the wiki also said that this is safe. And I was left with no way to run any other command.
Later I would find that I had libdl.so in /lib be a simple simlink to a file in /usr/lib. When the update happened, it should have been moved properly but no one treated this case. Since bash needs libdl.so nothing could be run. Moreover, /bin/sh was a simlink to /bin/bash, no busybox was installed. A bricked system.
To add more, this happened Sunday in the morning and I had no backup ISO at hand. The first day when I could make one was Monday, after 24 hours. A day of planned things was lost.
Due to some technical problems, the ISOs that I got on Monday were useless. Same for those of yesterday. Only today did I manage to install a temporary Linux Mint system and solve all the problems caused by the downgrade.
Yes, a downgrade because though Linux Mint says that it is a rolling release, it is not the same kind of rolling release as Arch. Chromium was several versions back and I had to spend an hour carefully moving files around in order to not lose a single piece of history, bookmark or passwords :)
On Arch, GHC was at least «The Glorious Glasgow Haskell Compilation System, version 7.0.3» (this data is from February). Now, I can only install a ghc6 package, clearly back in time. With the cabal cache created using a newer version I’m expecting new problems. Or, I could delete the entire cabal cache and reinstall from the start. Which way is recommended?
Now, I have this system but I don’t intend to keep it forever. Because it is not a rolling release as it should be. I don’t want old packages for stable things.
I want a system which can offer a newer version of a package as soon as possible after the software maintainers declare that version to be stable. A rolling release system, of course. But one which will not brick your system with experimental updates. Arch failed me on this and I’ll think twice before reinstalling it. Mint lies about rolling releases. What other systems can I use?
There are Sabayon and Gentoo but I’m not sure about going there (and I had some boot problems with both of them). And there is NixOS which is in an experimental state. Do you know another? Which one should I install?
So, are you using the Debian edition of Linux Mint?
I think a good solution to this would be the cloud sync option in Chromium. This should keep things in sync (with mobile phones and other Chromium devices too) and then you can safely delete config files and the like. That’s not as easy for other pieces of software though.
You see, I think the bricking problem is inherent in rolling release distributions, including the slower ones such as Debian Testing or Linux Mint (the Debian Edition) – or maybe just in Debian Unstable, I’m not too sure -, because this often requires changes in the package tree and/or the base system, and that’s very likely to cause problems.
I’m pondering myself whether having the latest stuff is actually worth the trouble. At some point all one needs is a working system and if a stable (even freeze-based) distro offers that then maybe it’s the better solution under the circumstances.
Oh, but on the other hand we’re supposed to be hackers and don’t mind the productivity issues caused by these updates at times.
Meh, I’ve never had a “bricked” install running Debian unstable, and I’ve been doing that for years, now. Debian is incredibly diligent about making sure their upgrades run reasonably clean. Has it been 100% problem free? No. But so long as you look at the package list before a dist-upgrade to make sure nothing you care about is going to get deleted or something, it’s rarely an issue.
Gah, I thought that that was your problem, but I hoped it wasn’t really.
The update is #1 issue that should be in the mind of a distribution maintainer, and the issue was solved a long time ago (even by breaking some of the rules to allow the update of glibc without using static executables). But this is something that should never ever be the issue of a user.
Now, staying up to date with the latest… well, even if some distributions offer that, I wouldn’t take it. New versions might mess up the system and downgrades are not an easy option in the world of Linux repositories. This reminds me of the issues that were brought up by Ingo Molnar, the guy who canceled the 2013 as the Linux Desktop Year: https://plus.google.com/109922199462633401279/posts/HgdeFDfRzNe
(As a joke, next year is traditionally the Linux Desktop Year, and that happens since 1998).
If you want some stuff to always update, take a bare, stable distribution, git clone the sources, configure, make, make install in /opt :)
Maybe NixOS would fit the bill:
http://nixos.org/nixos/
I’ve only read about it and haven’t used, but one of the features is
that older sets of packages are preserved and can be rolled back to
from GRUB.
I don’t know, but I’m hoping that an academic distribution with a
purely functional package manager may have attracted more attention
than the size of its community would otherwise warrant to having GHC
work on it as well :)
(Oops, you already mentioned NixOS. Sorry, I can’t read)
AFAIK Fedora was the first to move /lib/* -> /usr/lib. (At least of the Linux distros, I believe Solaris has done this long ago.)
Rationale can be found on: http://fedoraproject.org/wiki/Features/UsrMove
Personally, I had no problems during this Arch update. I did read the front page when the usual pacman -Syu failed though. The two steps shown there worked fine. Generally you are going to have these kinds of roadbumps for rolling releases — there are just too many combinations of circumstances and too few testing/QA resources to ensure that things’ll never break. If you want something more tested, I’d suggest a release-often distro instead.
The only distros that I’ve seen so far which really attempt to solve the “I want new packages but a stable base” problem properly are NixOS (Nix package manager) and PC-BSD 9.x (new PBI package format), but the BSDs seems to have lackluster (if any) support for KMS and NixOS is a little immature.
Thanks for the comments. It seems that I’m stuck with NixOS or Gentoo and derivatives. I’ll start testing them :)
IMHO Gentoo features the best support for haskell programming and provides tight integration of native package manager with cabal. Gentoo-haskell overlay always has the newest ebuilds, so you won’t be stuck back in time.
I started using NixOS over 2.5 years ago and I have never looked back.
But do you use it in production? On a real machine? Does it have everything you need?
I use NixOS on my laptop. It didn’t quite have everything I needed, but making (and contributing) packages wasn’t that hard. The selection is quite a bit better today than when I started.
Cool, thanks. I intend to switch to it after learning to use it :)
I would recommend against Gentoo – I had package breakages once a week for a while in 2010 through early 2011 (things like libpng broke ABI – it was fun, but Gentoo has great tools for finding and fixing the breakage). I also know people who use it in production and its a full time job to keep ahead of it for them (but that may be integrating their code into it/making sure there are no collisions). My debian system has had no problems, but i have only had it for a few months (I also run an Arch system and had problems with the update – I keep a rescue distro on my hd so it wasnt too bad).