I recently had to upgrade Ubuntu Xenial (16.04.5 LTS) to Bionic (18.04.1 LTS), and found a few pitfalls worth mentioning.
The server was a VPS from OVH, built from VPS 2016 SSD 1 template.
The first issue was that the upgrade process stuck at `Setting up libnih1:amd64 (1.0.3-6ubuntu2) …`
A quick investigation showed that the issue was `telinit u` run by libnih’s post-installation script:
For the upgrade to continue, `telinit` had to be killed.
The second issue was that `rsyslog` and `openssh-server` failed to upgrade properly.
Setting up rsyslog (8.32.0-1ubuntu4) ...
The user `syslog' is already a member of `adm'.
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Job for rsyslog.service failed because the control process exited with error code.
See "systemctl status rsyslog.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rsyslog, action "restart" failed.
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2018-08-26 18:35:54 UTC; 11ms ago
Docs: man:rsyslogd(8)
RSyslog Documentation
Process: 30924 ExecStart=/usr/sbin/rsyslogd -n (code=exited, status=1/FAILURE)
Main PID: 30924 (code=exited, status=1/FAILURE)
dpkg: error processing package rsyslog (--configure):
installed rsyslog package post-installation script subprocess returned error exit status 1
The log contains no useful information (except that `rsyslogd` failed to start), therefore we need to check manually what is wrong. `RSYSLOG_DEBUG=Debug rsyslogd -d` to the rescue.
In my case the issue is that the PID file exists, and it looks like it belongs to another rsyslogd process (though the PID is 372, and I have no idea why rsyslogd says it is 32050).
`systemctl stop rsyslog` obviously did not work, therefore I had to kill `rsyslogd` with `kill $(</run/rsyslogd.pid)`. After that `systemctl start rsyslog` worked:
root@ns2:~# systemctl start rsyslog
root@ns2:~# systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2018-08-26 18:55:30 UTC; 6s ago
Docs: man:rsyslogd(8)
RSyslog Documentation
Main PID: 454 (rsyslogd)
Tasks: 4 (limit: 2295)
CGroup: /system.slice/rsyslog.service
└─454 /usr/sbin/rsyslogd -n
сер 26 18:55:30 ns2.wildwolf.name systemd[1]: Starting System Logging Service...
сер 26 18:55:30 ns2.wildwolf.name systemd[1]: Started System Logging Service.
Now let us look what is wrong with OpenSSH:
Setting up openssh-server (1:7.6p1-4) ... Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details. invoke-rc.d: initscript ssh, action "restart" failed. ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2018-08-26 18:35:53 UTC; 9ms ago Process: 30742 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255) Process: 30741 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Main PID: 30742 (code=exited, status=255) сер 26 18:35:53 ns2.wildwolf.name sshd[30741]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication сер 26 18:35:53 ns2.wildwolf.name sshd[30741]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 16: Deprecated option UsePrivilegeSeparation сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 19: Deprecated option KeyRegenerationInterval сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 20: Deprecated option ServerKeyBits сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication сер 26 18:35:53 ns2.wildwolf.name sshd[30742]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication сер 26 18:35:53 ns2.wildwolf.name systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a сер 26 18:35:53 ns2.wildwolf.name systemd[1]: ssh.service: Failed with result 'exit-code'. сер 26 18:35:53 ns2.wildwolf.name systemd[1]: Failed to start OpenBSD Secure Shell server. dpkg: error processing package openssh-server (--configure): installed openssh-server package post-installation script subprocess returned error exit status 1
systemd was not very helpful as to what went wrong, therefore I had to run `sshd` manually to see what happens. The actual error was “Missing privilege separation directory: /run/sshd”:
I don’t know who is to blame: `systemd` (the error was not logged in the journal) or OpenSSH (it did no send the error to the system log), but in case of any issues with OpenSSH, `sshd -t` helps 🙂
Honestly, I failed to solve this issue: in theory, `systemd` itself should create `/run/sshd` because of `RuntimeDirectory=sshd` in `ssh.service`, but something went wrong. If I run `mkdir -m 0755 /run/sshd`, `systemctl start ssh` still fails because `/run/sshd` gets removed but not created. Luckily, I had a KVM, so I just rebooted the server. It came online with OpenSSH up and running.
The most funny issue was that Ubuntu offered to report those errors to developers; however, according to the installer, the problem cannot be reported because “This problem report is damaged and cannot be processed”. I cannot say I am impressed by the QA team 🙂
Over the years, Ubuntu LTS’s quality gets worse and worse. Unfortunately.