Vault 06 - Linux hardening
Vault Solution · Previous: Virtual machines and OS build · Next: TLS, DNS and certificates
A secrets store on a soft operating system protects nothing. Every server of the Solution was hardened against two standards at once: the organisation's own security baseline, which had to be signed off before go-live, and the CIS Benchmark for Oracle Linux 8, which gave the baseline its technical detail. This article covers what was applied, with what, and where the two standards were knowingly not followed.
The layers
| Layer | Tool | Where it is described |
|---|---|---|
| Kernel parameters | /etc/sysctl.conf | Below, and sysctl.conf |
| SSH server | /etc/ssh/sshd_config | Below, and sshd_config |
| CIS Benchmark | Ansible, RHEL8-CIS role | Below, and CIS hardening playbook |
| Mandatory access control | SELinux, enforcing | Virtual machines and OS build |
| Host firewall | firewalld, rich rules only | Network design and firewall flows |
| Audit trail | auditd, forwarded | Audit logging and log shipping |
| Continuous check | The organisation's scan client, daily | crontab |
Kernel parameters
The file combines network hardening with tuning for many connections. Today I maintain those two concerns as separate profiles, and they are the place to start for a new build:
What ran on these servers in 2023 is a predecessor of both. It is kept as a Config document, sysctl.conf, and differs from the maintained profiles in these points.
| Area | As built in 2023 | Maintained profile |
|---|---|---|
| Kernel self-protection | randomize_va_space=2, suid_dumpable=0 only | Adds kptr_restrict, dmesg_restrict, perf_event_paranoid, unprivileged BPF off, ptrace_scope, protected links, FIFOs and regular files |
| IPv6 | Disabled entirely, in sysctl and on the kernel command line | Left on, hardened |
tcp_timestamps | 0 | 1 |
| Socket buffers | 24 MB maximum and 24 MB default | 16 MB maximum, 256 KB default |
vm.swappiness | 0 | 10, or 1 for latency-sensitive hosts |
vm.dirty_ratio | 60 | 20 |
ip_local_port_range | 2048 65535 | 10240 65535 |
| ARP | arp_announce=2, arp_ignore=2 | Not part of the profile |
The only difference between roles is forwarding. A load-balancer node sets net.ipv4.ip_forward=1 and, in a separate file, net.ipv4.ip_nonlocal_bind=1, so that HAProxy can bind the virtual address while the other node holds it. Vault nodes and bastion hosts forward nothing.
SSH server
The same applies to SSH: the maintained profile is OpenSSH - Hardened SSHD configuration, and what ran here is its 2023 predecessor, kept as sshd_config.
| Area | As built in 2023, OpenSSH 8.0 | Maintained profile, OpenSSH 10 |
|---|---|---|
| Authentication | Public key only, no root login | The same |
| Who may log in | AllowUsers with named administrators and AllowGroups wheel, both required | AllowGroups with a dedicated group |
| Algorithms | Fixed lists for ciphers, key exchange, MACs and host keys | Left to the installed version, so new defaults such as post-quantum key exchange are not excluded |
| Host key | Ed25519 only | The same intent |
| Idle sessions | ClientAliveInterval 900, ClientAliveCountMax 0 | ClientAliveInterval 300, ClientAliveCountMax 2; since OpenSSH 8.2 a count of 0 disables the timeout |
| Forwarding | Everything off | The same |
On the bastion host three directives are reversed: AllowAgentForwarding yes, AllowTcpForwarding yes, DisableForwarding no. That is what makes it a jump host: an administrator logs in to the bastion and reaches every other server of the environment from there, and from nowhere else.
CIS Benchmark with Ansible
The benchmark was applied with the community RHEL8-CIS Ansible role, against CIS Oracle Linux 8 Benchmark v2.0.0. Ansible ran as the local recovery account, with one playbook per target host.
$ yum install python3-jmespath sshpass crypto-policies-scripts $ rpm -i ansible-2.9.27-1.el8.noarch.rpm $ ansible --private-key ~/.ssh/id_rsa --inventory /etc/ansible/hosts prod-vault-node1.example.net -m ping
A playbook names every rule of the benchmark and sets it to true or false. Nothing is left to the role's defaults, so the playbook is also the record of what was decided. It is a Config document: CIS hardening playbook.
The run was never one command. Each of the six benchmark sections was first run in check mode, the planned changes read, and then applied.
$ ansible-playbook --ask-become-pass --extra-vars "user=recovery ansible_distribution=RedHat ansible_distribution_major_version=8" ~/prod-vault-node1.example.net.yml -t section_1 --check $ ansible-playbook --ask-become-pass --extra-vars "user=recovery ansible_distribution=RedHat ansible_distribution_major_version=8" ~/prod-vault-node1.example.net.yml -t section_1
The two ansible_distribution variables are there because the role tests for Red Hat and Oracle Linux reports itself differently.
Deviations
Every rule that was not applied is listed with its reason, per host, in a compliance sheet that went to the security review. These are all of them.
| CIS rule | What it asks | Applied | Reason |
|---|---|---|---|
| 1.1.6.1 to 1.1.6.4 | Separate partition for /var/log/audit, with noexec, nodev, nosuid | No | Not required by the organisation's baseline; the image's volume layout was kept |
| 1.2.1 | Red Hat Subscription Manager configured | No | Not applicable to Oracle Linux |
| 1.4.1 | Bootloader password | No | Not required by the baseline; console access is controlled by the platform |
| 3.2.1 | IP forwarding disabled | Not on load balancers | Forwarding is needed there |
| 3.4.2.x | nftables rules | No | The host firewall is firewalld |
| 3.4.3.x | iptables and ip6tables rules | No | The host firewall is firewalld |
| 4.2.2.1.x, 4.2.2.5 | systemd-journal-remote | No | Remote logging is done with rsyslog |
| 5.2.13 | SSH TCP forwarding disabled | Not on the bastion | It is the jump host |
| Baseline SSH rule | SSH agent forwarding disabled | Not on the bastion | The same |
| 5.3.5, 5.3.6 | sudo re-authentication and timeout | Checked by hand | Already compliant in a form the role's test does not recognise |
| 6.2.9 to 6.2.11 | Home directories exist, are owned and are 750 or stricter | Not through the role | No reason is recorded in the sheet |
All optional server packages the role knows about, from a web server to a print service, are set to "not present".
Login through Active Directory
The notes contain a draft sssd.conf and krb5.conf for logging in with accounts of the organisation's Active Directory, with a simple access provider and an AD group for remote logon. The files still carry placeholders and the notes do not show it going further. The servers ran with local named accounts and SSH keys, as described in Virtual machines and OS build, and AuthenticationMethods publickey in the SSH configuration would not have admitted a directory password in any case.
The continuous check
Hardening decays. The organisation ran a central vulnerability and compliance scanner that works without logging in to the servers: a client on each server collects package versions and configuration facts once a day and uploads them, and the scanner reports missing patches and baseline violations per system. The client and its daily update are the first two jobs in the crontab of every server.
Reading it today
- The CIS role has moved on. The
RHEL8-CISrole follows newer benchmark versions with different rule numbers. A playbook that names every rule, as this one does, has to be rewritten for each benchmark version; that is its price. - Ansible 2.9 is long out of support. It was what the distribution's extra repository offered in 2023.
- Nothing here is specific to Vault 1.14. The current hardening guidance for Vault adds points this build already met, such as running as an unprivileged user, no swap or encrypted swap, and no core dumps, and one it did not: do not run other software on a Vault node. The snapshot agent, the AWS client and the monitoring agent all ran there.
- Production hardening