Vault 05 - Virtual machines and OS build

Vault Solution · Previous: Network design and firewall flows · Next: Linux hardening

Nineteen virtual machines, three kinds, one base build. This article covers how the machines were created and what was done to every one of them before any Vault software was installed.

The machines

The three roles come in two sizes.

Each environment also has one bucket in the private cloud's S3-compatible object storage, for snapshots and audit logs, billed by use.

The cloud template

The private cloud is built on VMware, and its self-service layer deploys from templates written in YAML. One template describes one environment: three existing networks, referenced by tag, and one machine resource per server with its image, size and static address.

  VM_PROD_VAULT_NODE1:
    type: Cloud.vSphere.Machine
    properties:
      image: <OL8_LVM_IMAGE>
      cpuCount: 4
      totalMemoryMB: 16384
      customizationSpec: Linux-selfmanaged
      networks:
        - address: 10.10.1.34
          gateway: 10.10.1.33
          netmask: 255.255.255.240
          network: '${resource.NET_PROD_VAULT_SERVICE.id}'
          assignment: static
      name: VM_PROD_VAULT_NODE1

The full file is a Config document: Cloud template.

The image is the platform's Oracle Linux 8 image with LVM, in the "self-managed" variant: the platform delivers a patched base system and stays out of it afterwards. Everything below was done by hand, from notes, one server at a time. The notes for one Vault node run to about 1,700 lines, and the five nodes of a cluster differ in a dozen of them.

Disk layout

The system disk arrives partitioned by the image.

/opt started at 2 GiB and was extended to 4 when the monitoring agent needed the room.

The data disk of a Vault node is added to the deployment after it exists, as a persistent disk, and prepared by hand.

$ fdisk /dev/sdb
$ pvcreate /dev/sdb1
$ vgcreate vg01 /dev/sdb1
$ lvcreate -l 100%FREE -n lv_data vg01
$ mkfs.ext4 /dev/vg01/lv_data
$ mkdir /data
$ mount -a
$ rmdir /data/lost+found

The matching line in /etc/fstab:

/dev/mapper/vg01-lv_data /data ext4 defaults,nodev,nosuid,noexec 0 2

/data is the Raft directory. lost+found is removed so that the directory holds nothing but what Vault puts there.

The base build

The same steps on every server, in this order.

Accounts

$ user=admin01
$ adduser $user
$ usermod -aG wheel $user
$ mkdir -p /home/$user/.ssh
$ touch /home/$user/.ssh/authorized_keys
$ chown -R $user:$user /home/$user/.ssh
$ chmod 700 /home/$user/.ssh
$ chmod 600 /home/$user/.ssh/authorized_keys

The administrator's public key goes into authorized_keys. The SSH server accepts public keys only and only for the named accounts, so the account's password matters for sudo and nowhere else.

Repositories

[ol8_baseos_latest]
name=ol8_baseos_latest
baseurl=http://10.40.2.10/repo-ol8/repo/ol8_baseos_latest
enabled=1
refresh=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle

ol8_appstream and ol8_addons have the same shape. The mirror is reached over plain HTTP inside the platform; package signatures are what is trusted, so gpgcheck=1 is not optional here. Vault nodes add the vendor's repository, which is on the Internet and reached through the platform's forward proxy.

Names and time

A Vault cluster whose nodes cannot resolve each other cannot elect a leader, and one whose clocks drift apart has trouble with certificates and leases. Both dependencies were made as local as possible.

The names the cluster needs are in /etc/hosts on every server of the environment: its own nodes, its load balancers and virtual address, the COMMON virtual address, the log collector and the object storage endpoints. DNS is the second source. The file is a Config document: /etc/hosts.

search example.net
nameserver 10.40.3.6
nameserver 10.40.4.6

That is /etc/resolv.conf. In /etc/chrony.conf three lines change.

server 10.40.3.6 trust
server 10.40.4.6
rtcsync

The public pool entry of the default chrony configuration is commented out; the servers have no route to it.

SELinux

The image does not arrive with SELinux enforcing. It is switched to enforcing with the targeted policy, the whole filesystem is relabelled on the next boot, and the result is checked.

$ yum install policycoreutils-python-utils
$ fixfiles -F onboot
$ init 6
$ getenforce
$ sestatus

Enforcing

From then on every file that is created by hand in a place SELinux cares about gets its context restored, which is why restorecon -RvF follows almost every step in the later articles. The one port outside the policy, the collector's TCP 50515, is added to the syslog port type.

$ semanage port -a -t syslogd_port_t -p tcp 50515

What differs by role

Reading it today

Doing this by hand nineteen times was the weakest part of the build. The notes were good enough that every server came out nearly the same, and "nearly" is what the firewall article and the differing Keepalived script paths in keepalived.conf show. The hardening was already run with Ansible; the base build should have been too, from one inventory with the per-node values in it.

Oracle Linux 8 is not named in the list of platforms in the current Vault documentation, which mentions RHEL and CentOS; the vendor's RHEL 8 package repository still carries current builds, 2.1.1 among them.

• Installing Vault