Ubuntu 24.04 upgrade report

Heya! I just upgraded to from Ubuntu 23.10 to Ubuntu 24.04 today, so I thought I'd publish a quick blog post on my experience. There are a number of issues to watch out for on this one.

tldr: Do not upgrade a machine to which you do not have physical access to 24.04 until the first point-release comes out!

While the do-release-upgrade itself went relatively well, I encountered a number of problematic issues that significantly affected the stability of my system afterwards, which I describe below, along with the fixes and workarounds that I applied.

(Above: One of the official wallpapers for Ubuntu 24.04 Noble Numbat entitled "Little numbat boy", drawn by azskalt in Krita)

apt sources

Of course, any do-release-upgrade you run is going to disable third-party sources. But this time there's a new mysterious format for apt sources that looks a bit like this:

Enabled: yes
Signed-By: /etc/apt/trusted.gpg.d/sbrl.asc
Types: deb
URIs: https://apt.starbeamrainbowlabs.com/
Suites: ./
Components: 

....pretty strange, right? As it turns out, Ubuntu 24.04 has decided to switch to this new "DEB822" apt sources format by default, though I believe the existing format that looks like this:

deb [signed-by=/etc/apt/trusted.gpg.d/sbrl.asc] https://apt.starbeamrainbowlabs.com/ ./ # apt.starbeamrainbowlabs.com

....should still work. Something else to note: the signed-by there is now required, and sources won't work without it.

For more information, see steeldriver's Ask Ubuntu Answer here:

Where is the documentation for the new apt sources format used in 24.04? - Ask Ubuntu

Boot failure: plymouth and the splash screen

Another issue I encountered was this bug:

boot - Kubuntu 24.04 Black Screen / Not Booting After Upgrade - Ask Ubuntu

...basically, there's a problem with the splash screen which crashes the system because it tries to load an image before the graphics drivers load. The solution here is to disable the splash option in the grub settings.

This can be done either before you reboot into 24.04, or if you have already rebooted into 24.04, in the grub menu you can simply hit e on the default Ubuntu entry in your grub menu and then remove the word splash from the boot line there.

If you are lucky enough to see this post before you reboot, then simply edit /etc/default/grub and change quiet splash under GRUB_CMDLINE_LINUX_DEFAULT to be an empty string:

GRUB_CMDLINE_LINUX_DEFAULT=""

...and then update grub like so:

sudo update-grub

Boot failure: unable to even reach grub

A strange one I encountered was an inability to even reach grub, even if I manually select the grub.efi as a boot target via my UEFI firmware settings (I'm on an entroware laptop so that's F2, but your key will vary).

This one kinda stumped me, so I found this page:

Boot-Repair - Community Help Wiki

...which suggests a boot repair tool. Essentially it reinstalls grub and fixes a number of other common issues, such as a missing nvram entry for grub (UEFI systems need bootloaders registering against them), missing packages - I suspect this was the issue this time - and other common issues.

It did claim that my nvram was locked, but it still seems to have resolved the issue anyway. I do recommend booting into the live Ubuntu session with the toram kernel parameter (press e in the grub menu → add kernel parameter → press ctrl + x) and them removing your flash drive before running this tool, just to avoid it getting confused and messing with the bootloader on your flash drive - thus rendering it unusable - by accident.

Essentially, boot into a live environment, connect to the Internet, and run then these commands:

sudo add-apt-repository ppa:yannubuntu/boot-repair && sudo apt update
sudo apt install -y boot-repair
boot-repair

sudo is not required for some strange reason.

indicator-keyboard-service memory leak

Finally, there is a significant memory leak in indicator-keyboard-service - which I assume provides the media/function key functionality, which I only noticed because I have a system resource monitor running in my system tray (indicator-multiload; multiload-ng is an alternative version that may work if you have issues with the former).

The workaround I implemented was to move the offending binary aside and install a stub script in its place:

cd /usr/libexec/indicator-keyboard
sudo mv indicator-keyboard-service indicator-keyboard-service.bak
sudo nano indicator-keyboard-service

In the text editor for the replacement for indicator-keyboard-service, paste the following content:

#!/usr/bin/env sh
exit 0

...save and exit. Then, chmod +x:

sudo chmod +x indicator-keyboard-service

....this should at least workaround the issue so that you can regain system stability.

I run the Unity desktop, but this will likely affect the GNOME desktop and others too. There's already a bug report on Launchpad here:

Bug #2055388 "suspected memory leak with indicator-keyboard (causing gnome-session-flashback to freeze after startup)" : Bugs : indicator-keyboard package : Ubuntu

...if this issue affects you, do make sure to go and click the green text at this top-ish of the page to say so. The more people that say it affects them, the higher it will be on the priority list to fix.

Conclusion

A number of significant issues currently plague the upgrade process to 24.04:

• Memory leaks from indicator-keyboard-service
• Multiple issues preventing systems from booting by default

...I recommend that upgrading to 24.04 is done cautiously at this time. If you do not have physical access to a given system or do not have the time/energy to fix issues that prevent your system from booting successfully, I strongly recommend waiting for the first or second point release (i.e. 24.04.1 / 24.04.2) before upgrading.

If you haven't already, I also strongly recommend configuring timeshift to take automated snapshots of your system so that you can easily roll back in case of a failure.

Finally, I also recommend upgrading via the command line with this command:

sudo do-release-upgrade

...and carefully monitoring the logs as the upgrade process is running. Then, do not reboot as it asks you to until you have checked and resolved all of the above issues.

That's all I have at the moment for upgrading Ubuntu. I have 3 other systems to upgrade from 22.04, but I'll be waiting for the first point release before attempting that. I'll make another post (or a comment on this one) to let everyone know how it went when I do begin the process of upgrading them.

If you've encountered any issues in the upgrade process to 24.04 (or have any further insight into the issues I describe here), please do leave a comment below!

Ubuntu 22.04 upgrade report

(Above: A slice of the official Ubuntu 22.04 Jammy Jellyfish wallpaper)

Hey there! Since Ubuntu 22.04 Jammy Jellyfish has recently been released, I've upgraded multiple machines to it, and I have enough to talk about that I thought it would be a good idea to write them up into a proper blog post for the benefit of others.

For reference, I've upgraded my main laptop on 20th May 2022 (10 days ago as of writing this psot), and I've also upgraded one of my desktops I use at University. I have yet to upgrade starbeamrainbowlabs.com - the server this blog post is hosted on - as I'm waiting for 22.04.1 for that (it would be very awkward indeed if the upgrade failed or there was some other issue I'm not yet aware of).

The official release notes for the Ubuntu 22.04 can be found here: https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668

There's also an official blog post that's ranked much higher in search engines, but it's not really very informative for me as I don't use the GNOME desktop - you're better off reading the real release notes above.

Thankfully, I have not encountered as many issues (so far!) with this update as I have with previous updates. While this update doesn't seem to change all that much aside from a few upgrades here and there, by far the biggest annoyance is shipping Firefox as a snapd by default.

Not only are they shipping it as a snap package, but they have bumped the epoch number, which means that the packages in the official firefox apt repository (beta users like me, use this one instead) are ignored in favour of the new snap package! I mean I get that shipping packages simplifies build systems for large projects like Firefox, but I have a number of issues with snapd:

• Extra disk space usage: every snap package has it's own version of it's dependencies
• Permissions: as far as I'm aware (please comment below if this is now fixed), there are permissions issues if you try to load a file from some places on disk when you're running an app installed via snapd, as it runs in a sandbox (this is also true of apps installed with flatpak). This makes using most applications completely impractical
• Ease of updates: A minor annoyance, but with apps installed via snap I have 2 different package managers to worry about
• Observability: Another minor concern, but with every package having it's own local dependencies, I'm makes it more difficult to observe and understand what's going on, and fix any potential issues

This aside, apt does allow for pinning apt repositories to work around this issue. I'll be posting a blog post on how this works more generally hopefully soon, but for now, you want to put this in a file at /etc/apt/preferences.d/firefox (after installing one of the above 2 apt repositories if you haven't done so already):

Package: *
Pin: release o=LP-PPA-mozillateam
Pin-Priority: 1001

...then run this sequence of commands:

sudo apt update
sudo apt purge firefox # This will *not* delete your user data - that's stored in your local user profile 
sudo apt install firefox

The above works for both the stable and beta versions. Optionally: sudo apt purge snapd.

I also found this necessary for the wonderful nautilus-typeahead apt repository.

This was the most major issue I encountered. Other than this, I ran into a number of little things that are worth noting before you decide to upgrade. Firstly, for those who dual (or triple or even more!) boot, the version of the grub bootloader shipped with Ubuntu does not detect other bootable partitions!

Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.

....so if you do run more than a single OS on your system, make sure you correct this after upgrading.

Another thing is that, as usual, Ubuntu disables all third party apt repositories on upgrade. I strongly recommend paying very close attention to the list of packages that do-release-upgrade decides it needs to remove, as if you install e.g. Inkscape or Krita via an apt repository to get the latest versions thereof, you'll need to reinstall them after re-enabling your apt repositories. Personally, I say "no" to the reboot at the end of the upgrade process and fix my apt repositories before then running:

sudo apt update
sudo apt dist-upgrade
sudo apt autoremove
sudo apt autoclean
# See also https://gitlab.com/sbrl/bin/-/blob/master/update-system

...and only then rebooting.

While GitHub's Atom seems to be more and more inactive these days as people move over to Visual Studio Code, I still find myself using it regularly as my primary code editor. Unfortunately, I encountered this bug, so I needed to edit /usr/share/applications/atom.desktop to add --no-sandbox to the execution line when starting Atom. The Exec= line in that file now reads:

Exec=env ATOM_DISABLE_SHELLING_OUT_FOR_ENVIRONMENT=false /usr/bin/atom --no-sandbox %F

This issue only occurred on 1 of the 2 systems I've upgraded though, so I'm not sure of the root cause. Other random issues I encountered:

• GDM has a truly awful shade of grey in the background now. This repository gives a way to fix this problem. Try to avoid an image that's too light in colour, as the white text of the lock screen becomes rather difficult to see.
• Speaking of backgrounds, the upgrade reset my desktop background on both machines I upgraded. Make sure you have a copy of it stored away somewhere, as you'll need it. lightdm (the login screen I use on my main laptop in place of gdm) seems to be fine though.
• tumbler - a d-bus thumbnailing service - was also automatically removed. This does not appear to have caused me any problems so far (though image previews now make transparent pixels appear white, which is really annoying and I haven't yet looked into a fix on that one), so I need to look into this one further.
• If you're a regular user of Memtest86+, it may disappear from your grub bootloader menu if you use EFI boot now for some strange reason.
• The colour scheme in the address bar of Nautilus (the file manager) seems a bit messed up for me, but this may have more to do with the desktop theme I'm using.

If I encounter any other issues while upgrading my servers in the future, I'll make another post here about it if it's a significant issue, or comment on/edit this post if it's a minor thing.

If you encounter any other issues upgrading that aren't mentioned here, please do leave a comment below with the issue you encountered and the solution / workaround you implemented to fix it.

Achievement Get: Upgrade Server - A writeup of my experiences

The upgrade is complete! I've managed to move practically everything over to the new server, apart from a few automated cleanup tasks here and there. There are still a few issues floating around, but they shouldn't affect this website, and I should have them cleared up soon. Although the migration went smoother than I expected, I did encounter some issues and learnt a few things that I thought I'd share here.

The first thing I found was that starting a todo list isn't a rather good idea. It sounds simple, but it's actually really useful. I found that I had a lot of small tasks I needed to complete, and I kept thinking of more things that needed doing at regular intervals. If I didn't write them down I'd never get anything done because I'd never be able to remember what needed doing first!

It also helps to do your research before you move. Make sure that you're properly reaquainted with all the software running on the system that you're going to migrate from, and that you're familiar with all the ins and outs of your particular situation. If you aren't, then you risk stumbling across some rather nasty, complicated, and time consuming problems mid-migration.

It also helps to do as much of the migration as possible without taking the old system offline. Install the software, Move the configuration files. Set up the firewall. Set up your new monitoring tools. This allows you to minimise the downtime that you have to subject your users to, which is always a good thing.

Lastly, testing is incredibly important. test everything. Make sure that every little feature after you migrate. You'd be surprised at how many issues can crop up after migration.

Server migration!

I've been watching Kimsufi's server page for what feels like absolutely ages now, waiting to get my hands on an ultra-cheap €4.99 per month (excluding VAT of course) KS-1 dedicated server. Unfortunately I've never been quite fast enough, so yesterday I decided that enough was enough and went ahead and bought a KS-2B at €9.99 per month (again excluding VAT). After all is said and done it works out to about £8.39 per month, which, for 2 cores / 4 threads, 4GB RAM, and a 40GB SSD, is an absolute bargain in my eyes.

I've been busy moving things across and it's going well, but I haven't finished yet - I still have the web server and the mail server to set up. I'm also looking at using the Hiawatha webserver instead of Nginx. Hiawatha is a security-first and easy to configure web server. Apparently it's also lightweight, but we'll see about that...! Nginx's config files have been annoying me for a while now, so I think that it's high time I tried something different.

3 Things to do before you upgrade to Ubuntu 16

Since Ubuntu 16.04 long term support has been released, I naturally have been wanting to upgrade to it. However, due to one thing or another, it hasn't been going so well. I decided to put this post together so that you don't make the same mistakes I did. Most of these tips I'm going to share with you in this post can be applied to be major upgrade process, although I found them useful when upgrading my Laptop to Ubuntu 16.04.

Make a backup. Seriously. I can't stress this enough. If the upgrade process gets interrupted or any reason, you'll be left with an unstable system - and a restore will be practically your only option.

Personally I use Timeshift to keep snapshots of my operating system files, and the default Déjà Dup backup program to backup my files. That way I can restore my OS to an earlier state without touching my files at all. Similarly, I can search though the history of my files without the clutter of a bunch of system files getting in the way.

Run the upgrader in either screen or tmux. Screen or Tmux let you run commands in a virtual window, and reattach to that window later if you loose your connection unexpectedly. In my case, I managed to lose access to the upgrader script halfway through the upgrade process, forcing me to kill the process and start over.

Make a checklist of things to do / test after the upgrade. Whilst upgrading, you'll find you start building up a long list of things you want to do and test after the upgrade is finished, such as reactivating extra PPAs, or merging config files. I recommend you write them all down somewhere so that you don't forget something.

That's about all I can think of at the moment. What upgrade are you in the process of doing? Do you have any tips to share? Write a comment below!

Resources

• Backup your system - Ubuntu community wiki
• How to use Linux Screen - Rack aid

Upgrading to Ubuntu 15.04 Vivid Vervet

Hello!

Yesterday you probably noticed some downtime. This is because I was upgradting this server's operating system from Ubuntu 14.10 to Ubuntu 15.04! Since I noticed a few things that you should watch out for when upgrading, I thought that I would make a post about it.

For the most part the upgrade went smoothly, but I did hit a few snags. Firstly, if you have got any custom ppas set up for apt-get, you will want to make a list of them (they are located in /etc/apt/sources.list.d) because the upgrade will annoyingly disable them all :( It's not too much trouble to fix but it is annoying.

Secondly, there are two new mime types that have been added /etc/mime.types. If you have made any customisations to this file (I have added text/x-markdown), then you will want to make a note of them and re-add them afterwards. Don't forget to restart your http servers after changing it!

There are some changes that require the ssh daemon to be stopped, so make sure you don't do the upgrade over ssh.

You will get asked which interfaces DCHPv6 should listen on / send requests to. If you use your linux box as a router and have it handing out IP addresses, then you will need to take note of which interfaces you have on your box and which one is which.

By far the biggest problem for me though was the switch from upstart to systemd. This server is hosted by OVH under one of their VPS hosting plans (which are great by the way!), which means that it is virtualised using OpenVZ. It also means that I can't choose my kernel :( I suspect that this is the reason that I can't use systemd, though if anyone has any other ideas, I would love to hears them - leave them in the comments below. When it has finished the upgrade, it couldn't reboot, instead telling me that it couldn't find an alternative telinit implementation to spawn. The solution to this is simple though (don't forget to run as root):

apt-get install upstart
apt-get remove systemd
apt-get install upstart-sysv

The last package in the above (upstart-sysv) should be install automatically, but you should make sure that it is installed - it is the package that prevents it from automatically trying to switch you back to systemd at the next available opportunity.

I hope this post is useful! If you do find it helpful, please leave a comment. If people seem to like it I might start posting full upgrade guides.