Upgrading All The Things to Ubuntu 22.04

One of my "winter break" projects this year was to get all of my disparate Ubuntu server instances upgraded and into parity. Last year I wrote about my adventure moving WSL Ubuntu from 18.04 to 20.04, which happened before 22.04 was officially released. In that process I noted the longer-term target of moving to 22.04 which brings us to the here and now!

It "Was Just Yesterday"

While it felt like that, the reality is my move to 22.04 was primarily driven by my desire to move to PHP 8.1. Ubuntu 20.04 doesn't include PHP 8 in its default repositories. Sure, I could use the PPA for all the PHP releases but I try to keep things as vanilla as possible across environments and only deviate as necessary (should a specific or newer version of PHP be required for a project). This keeps things easily replicable ("out of the box").

Further, since the last upgrade of WSL Ubuntu, I added a couple server instances to my little fleet, some of which are on real hardware (not virtual machines). Those machines were already running 22.04, but some of my other VMs out in the world were originally spun up as 20.04 instances. I had a whole mix of OS versions.

Performing the Upgrade

Much like the previous move, the actual upgrade wasn't too difficult. Running the "prestaging" commands to get things ready to go was pretty routine:

sudo apt update
sudo apt list --upgradable
sudo apt upgrade
sudo apt --purge autoremove
sudo apt install update-manager-core

Don't Forget to Back Up Important Servers First!

This was especially important for my Internet-facing instances, which are fortunately all virtual machines. Creating a pre-upgrade snapshot or otherwise backing up important bits is a step you definitely don't want to forget because things can go sideways...

I didn't need full backups for other instances, so some creative tarballing was in order. This process of thinking out backups leads to my next post that will identify how I set up my "new" backup structure and mechanisms. Before this upgrade cycle I had no "automatic" backups in place for these servers/services—the process of identifying the important bits was important in getting to the "last mile" of such a process.

The Ubuntu 22.04 Upgrade Itself

The upgrade itself is triggered with the following command:

sudo do-release-upgrade

Different machines upgraded at different speeds, but most completed within 15 minutes or so. There are several prompts along the way with config file diffs...paying attention to those and understanding them is a good idea. In most cases, I kept my previous configurations but each situation is unique. Ultimately, following the upgrade and after a final reboot, I was brought to some version of the welcome message:

Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-58-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of <date string>

  System load:  0.0                Processes:             112
  Usage of /:   0.9% of 250.98GB   Users logged in:       1
  Memory usage: 3%                 IPv4 address for eth0: <private ipv4>
  Swap usage:   0%

 * Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
   just raised the bar for easy, resilient and secure K8s cluster deployment.

   https://ubuntu.com/engage/secure-kubernetes-at-the-edge

0 updates can be applied immediately.

Not Everyone Was Happy, Though

One VM had some sort of "problem" during the upgrade. I don't know exactly what happened, but in the process of rebooting during the upgrade sequence all sorts of stuff went sideways, and the box was essentially "bricked." Not even console access was behaving as I'd expect, so it was back to the snapshot/backup.

This is why backups are important!

Same box after restoration and a second attempt for an upgrade: no major issues; the box upgraded as expected. It's still periodically misbehaving with regard to network, though, and I encounter an odd and very infrequent problem where randomly after a reboot the network won't behave. A second reboot (via the console) solves the problem. At first I thought it might be firewall related, but temporarily disabling firewall had no impact. Only a second reboot has consistently fixed the situation.

To that end this particular VM has a slightly different network configuration than my others (multiple static IPs for one interface). Otherwise the box configuration is the same, so I think this might be related to something transient with routing. The box always has a link/appears "up," so it doesn't think it has a network problem...yet it does.

At this point I think there could be something wonky with the netplan, but the box is so infrequently rebooted (only for kernel updates) I haven't had the problem persist enough to really dig into it. I might try to fix it, but I might also just nuke it and start afresh some weekend. In the meantime I more deliberately plan when to reboot that particular machine and know I might need to jump in at the console to solve any hiccups.

Everyone's Happy in the End

Hiccup aside, all of these machines are now in alignment, on 22.04, and running PHP 8.1. I have had to leverage the PHP PPA on one server, though, due to a specific version requirement for a project I'm working on. That box is also enabled for multiple PHP versions, so in a weird way it's not terribly out of line.

It's nice to have everyone in the same rough configuration. Maybe this is a first sort of stepping stone to using something like Ansible to manage configurations...we'll see. ¯\_(ツ)_/¯

Even better, this upgrade process forced me to come up with a more realistic backup scheme for all of these moving parts!