How to Make System Logs Append-Only

During an intrusion, an intruder leaves signs of his actions in various system logs. Without reliable logs, it could be very difficult to figure out how the attacker got in, or where the attack came from. This information is crucial in analyzing the incident. It is evident that the logs are a valuable audit trail that should be well protected.

Of course, when an intruder gets in to the system, they will try to remove all traces. So, how can we stop an intruder from removing evidence?

One of the options is to send all logs to a remote server. This option is definitely worth considering because when a root is compromised, it all depends on how knowledgeable and experienced the intruder is.

However, when you cannot use a remote server for storing logs, or as an additional line of defense, there is another option: make critical log files append-only. This is where chattr comes to help.

When a file has the “append-only” attribute set (chattr +a), that file cannot be deleted, and writes are only allowed to append to the end of the file. In other words, you cannot remove lines from the middle of the file.

For example,

root@server:~# chattr +a /var/log/auth.log
root@server:~# echo TEST > /var/log/auth.log
bash: /var/log/auth.log: Operation not permitted

Now we need to configure logrotate so that it can rotate log files. To do that, we need to remove append-only attribute from the file before it gets rotated and restore it afterwards.

On my Ubuntu machine, logrotate configuration for /var/log/auth.log looks like this:

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
}

What we need to change is to add a prerotate action which will remove append-only attribute, and a postrotate one to add it back.

Like this:

prerotate
    /usr/bin/chattr -a /var/log/auth.log
    /usr/bin/chattr -i /var/log/auth.log.*
endscript
postrotate
    /usr/bin/chattr +a /var/log/auth.log
    /usr/bin/chattr +i /var/log/auth.log.*
    /usr/lib/rsyslog/rsyslog-rotate
endscript

I prefer to use “immutable” attribute on the old log files to protect them from tampering.

OK, but if we can use chattr -a or chattr -i to remove the “append-only” or “immutable” attributes, why cannot the intruder do the same?

Let us read the manual page for chattr again:

A file with the ‘a’ attribute set can only be open in append mode for writing. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

A file with the ‘i’ attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

So the key is the CAP_LINUX_IMMUTABLE capability. If we somehow remove this capability, the intruder, even if they are root, will be unable to remove the attributes.

The most common way to gain root privileges is either to brute force the password to the root account, or to exploit a vulnerability in a daemon / process running as root.

systemd has a useful option, CapabilityBoundingSet. The option controls which capabilities to include in the capability bounding set for the executed process. What does this mean for us? If we edit ssh.service with systemctl edit ssh and add the following piece of code:

[Service]
CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE

and then restart ssh (systemctl restart ssh) and then log in as root, we will no longer be able to change append-only and immutable attributes:

root@server:~# capsh --print | grep imm
root@server:~# chattr -a /var/log/auth.log
chattr: Operation not permitted while setting flags on /var/log/auth.log

To secure the system further, you will need to make /etc/systemd/system/ssh.service.d/override.conf immutable as well (otherwise the intruder may delete that file, restart ssh and login again). However, even this can be not enough: if the intruder is smart enough, they can still circumvent this.

Even though MAC/RBAC is probably the only reliable way to protect the system, the described method can still be used as an additional line of defense.