Pitfalls When Upgrading Ubuntu Xenial to Bionic

I recently had to upgrade Ubuntu Xenial (16.04.5 LTS) to Bionic (18.04.1 LTS), and found a few pitfalls worth mentioning.

The server was a VPS from OVH, built from VPS 2016 SSD 1 template.

The first issue was that the upgrade process stuck at Setting up libnih1:amd64 (1.0.3-6ubuntu2) ...

A quick investigation showed that the issue was telinit u run by libnih’s post-installation script:

For the upgrade to continue, telinit had to be killed.

The second issue was that rsyslog and openssh-server failed to upgrade properly.

Setting up rsyslog (8.32.0-1ubuntu4) ...
The user syslog' is already a member of '.
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Job for rsyslog.service failed because the control process exited with error code.
See "systemctl status rsyslog.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rsyslog, action "restart" failed.
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2018-08-26 18:35:54 UTC; 11ms ago
     Docs: man:rsyslogd(8)
           
RSyslog Documentation

  Process: 30924 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=1/FAILURE)
 Main PID: 30924 (code=exited, status=1/FAILURE)
dpkg: error processing package rsyslog (--configure):
 installed rsyslog package post-installation script subprocess returned error exit status 1

The log contains no useful information (except that rsyslogd failed to start), therefore we need to check manually what is wrong. RSYSLOG_DEBUG=Debug rsyslogd -d to the rescue.

In my case the issue is that the PID file exists, and it looks like it belongs to another rsyslogd process (though the PID is 372, and I have no idea why rsyslogd says it is 32050).

systemctl stop rsyslog obviously did not work, therefore I had to kill rsyslogd with kill $(</run/rsyslogd.pid). After that systemctl start rsyslog worked:

root@ns2:~# systemctl start rsyslog
root@ns2:~# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-08-26 18:55:30 UTC; 6s ago
     Docs: man:rsyslogd(8)
           
RSyslog Documentation

 Main PID: 454 (rsyslogd)
    Tasks: 4 (limit: 2295)
   CGroup: /system.slice/rsyslog.service
           └─454 /usr/sbin/rsyslogd -n

сер 26 18:55:30 ns2.wildwolf.name systemd[1]: Starting System Logging Service...
сер 26 18:55:30 ns2.wildwolf.name systemd[1]: Started System Logging Service.

Now let us look what is wrong with OpenSSH:

Setting up openssh-server (1:7.6p1-4) ...
Job for ssh.service failed because the control process exited with error code.
See "systemctl status ssh.service" and "journalctl -xe" for details.
invoke-rc.d: initscript ssh, action "restart" failed.
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2018-08-26 18:35:53 UTC; 9ms ago
  Process: 30742 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
  Process: 30741 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 30742 (code=exited, status=255)

сер 26 18:35:53 ns2.wildwolf.name sshd[30741]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication
сер 26 18:35:53 ns2.wildwolf.name sshd[30741]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication
сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 16: Deprecated option UsePrivilegeSeparation
сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 19: Deprecated option KeyRegenerationInterval
сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 20: Deprecated option ServerKeyBits
сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication
сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication
сер 26 18:35:53 ns2.wildwolf.name systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
сер 26 18:35:53 ns2.wildwolf.name systemd[1]: ssh.service: Failed with result 'exit-code'.
сер 26 18:35:53 ns2.wildwolf.name systemd[1]: Failed to start OpenBSD Secure Shell server.
dpkg: error processing package openssh-server (--configure):
 installed openssh-server package post-installation script subprocess returned error exit status 1

systemd was not very helpful as to what went wrong, therefore I had to run sshd manually to see what happens. The actual error was “Missing privilege separation directory: /run/sshd”:

I don’t know who is to blame: systemd (the error was not logged in the journal) or OpenSSH (it did no send the error to the system log), but in case of any issues with OpenSSH, sshd -t helps 🙂

Honestly, I failed to solve this issue: in theory, systemd itself should create /run/sshd because of RuntimeDirectory=sshd in ssh.service, but something went wrong. If I run mkdir -m 0755 /run/sshd, systemctl start ssh still fails because /run/sshd gets removed but not created. Luckily, I had a KVM, so I just rebooted the server. It came online with OpenSSH up and running.

The most funny issue was that Ubuntu offered to report those errors to developers; however, according to the installer, the problem cannot be reported because “This problem report is damaged and cannot be processed”. I cannot say I am impressed by the QA team 🙂

Over the years, Ubuntu LTS’s quality gets worse and worse. Unfortunately.