Email 04 - Virtual machines and OS build

Email Solution · Previous: Network, DNS and firewall · Next: Internal servers: Postfix

Five small virtual machines per site carry the whole email system. This part covers where they run, how big they are, what the operating system template gave them and what had to be added before Postfix was configured: packages, two local accounts, a kernel parameter and three local SELinux modules. The SELinux part is the longest, because every one of those servers ran in enforcing mode and three daemons did things the stock policy does not expect.

Placement

The constraint from the design was short: virtual machines, on Red Hat Enterprise Linux 7. Each site has its own RHV installation with hypervisors in two datacenters, A and B, and the letter in a server name says where the machine is meant to run. High availability of the email service is done by the applications (a virtual address for the internal pair, a fallback relay), so the only thing asked of the virtualization platform is to keep the two halves apart.

flowchart LR
  subgraph dca["Datacenter A, DC2-A-CKVM001 to 004"]
    x1["DC2-A-VCMSX001, internal, Keepalived master"]
    x2["DC2-A-VCMSX002, mailboxes and webmail"]
    r1["DC2-A-VCMSR001, relay, primary"]
  end
  subgraph dcb["Datacenter B, DC2-B-CKVM001 to 004"]
    x3["DC2-B-VCMSX001, internal, Keepalived backup"]
    r2["DC2-B-VCMSR001, relay, fallback"]
  end
  x1 <-- "VRRP, virtual address" --> x3
  x1 -- "relayhost" --> r1
  x1 -. "fallback relay" .-> r2

The design asks for affinity rules on RHV that tie each machine to the four hypervisors of its datacenter.

The cell for the mailbox server is empty in the design document. Its name puts it in datacenter A, and since it is a single machine with no partner there is nothing to keep it apart from. The table for site 1 is the same with DC1 in every name. The Source material holds the rules as a requirement ("should be configured") and nothing from RHV itself, so I cannot show how they were entered.

Sizing

All five servers got the same compute, and only the mailbox server got a data disk.

The first concept had one internal server per site that did everything: SMTP, mailboxes and webmail. When the roles were split over three machines in 2019, each machine was given half of what the single one had. The mail is notifications and alerts from servers and appliances, and the design holds no figures for volume; the sizing was a starting point, not the result of a measurement.

File system layout

The design gives one layout for all servers, the standard one of the virtual machine template.

The fstab files of the five site 2 servers agree with the mount points and disagree with the table in the details, because the machines come from two generations of the template.

The dates are the "Created by anaconda" lines of the files. Two servers were installed in 2017, in the time of the first concept with one internal server and one relay per site, and were kept and reconfigured; the three from February 2019 were added for the HA concept. The older template mounted /home, /tmp and /var with nosuid,noexec,nodev; the newer one mounts everything with defaults. So the two nodes of the internal pair, which should be twins, differ in the hardening of their file systems. Nobody decided that; it came with the template.

The data disk of the mailbox server is one line.

/dev/mapper/vg01-data   /data                   ext4    defaults        1 2

The logical volume is data in vg01, not lv_data as in the design, following the naming of the newer template. The install notes start with /data already mounted; the commands that created the volume group and the file system are not in them.

Operating system and base services

Red Hat Enterprise Linux 7, deployed from the default template, with "the default security baseline for OS images" as the older design documents put it. The template brings the things every server of the management infrastructure has: sshd, sssd for logins from Active Directory, ntpd, rsyslogd sending to the central syslog, auditd, the Sensu client, etckeeper. They come back in Accounts, clients and operations. Four of the seven archived ifcfg files carry a ## Ansible managed header, so the template did not end at the image.

Time. ntpd with two server lines. According to the design, DNS and NTP for both sites come from the Infoblox appliances in site 1. The file is a Config document: ntp.conf.

Name resolution. The same resolv.conf on all five servers, and it shows that site 2 has no DNS servers of its own: the two IPv4 name servers are the site 1 Infoblox addresses.

search oob.example.net pc.example.net adm.example.net cimc.example.net mgmt.example.net pb.example.net
options timeout:1 attempts:1 rotate
nameserver 10.11.16.145
nameserver 10.11.18.145
nameserver 2001:db8:a1:c0b::f:1
nameserver 2001:db8:b1:c0a::f:1

One second and one attempt per server, with rotation, is a sensible setting for a mail server: Postfix does a lookup for nearly everything, and a resolver that waits five seconds for a dead name server stalls the queue. What I see today is that a pre-production mail system whose every lookup crosses to the production site is not independent of it.

Package repositories. The last step in every install note turns off the repository file managed by the subscription manager, so that only the repositories set by the build are used. It ran as root.

$ subscription-manager config --rhsm.manage_repos=0

Kernel parameters. One file, on the internal pair only. Postfix on both nodes sends from the virtual address, and the node that does not hold the address must still be able to bind to it. The file is a Config document: sysctl.d/postfix.conf. The install notes activate it with a reboot.

$ vi /etc/sysctl.d/postfix.conf
$ init 6

Software packages

The design has a table of packages per server. The install notes show what was typed, and that is a little more.

The table in the design lists postfix for all five, dovecot for the three internal servers, and the web stack for the mailbox server, with four PEAR components beside it: Mail_Mime 1.10.1, Net_IDNA2 0.2.0, Net_SMTP 1.8.0 and Auth_SASL. It leaves out Keepalived, GnuPG, rsync and inotify-tools, which the HA concept and the filter cannot work without. Roundcube comes from EPEL, everything else from the distribution.

On every server the stock configuration was copied aside and the live directory emptied before my own files went in. The commands ran as root; the relays did the same for Postfix only. On the internal pair the stock configuration of Keepalived was copied aside as well (mkdir and cp -R), but its directory was not emptied.

$ yum install postfix dovecot
$ systemctl stop postfix
$ systemctl stop dovecot
$ mkdir -p /etc/postfix-original
$ mkdir -p /etc/dovecot-original
$ cp -R /etc/postfix/* /etc/postfix-original/
$ cp -R /etc/dovecot/* /etc/dovecot-original/
$ rm -rf /etc/postfix/*
$ rm -rf /etc/dovecot/*

Local accounts

Postfix, Dovecot, Apache and MariaDB run under the accounts their packages create. Two more were made by hand.

The commands, as root. On the internal pair the note beside the vmail account says "not used on this server, for future use", because no mail is stored there.

$ groupadd -g 501 vmail
$ adduser --system --home /data/vmail --uid 501 --gid 501 --shell /sbin/nologin vmail
$ useradd bash-postfix-encrypt-filter

The install notes create the filter account without a number, and /etc/passwd on both nodes has 7778 for user and group. A plain useradd would not arrive at the same unusual number on two servers built two years apart, so the number was given by hand or by the build tooling, and the notes are incomplete here. The filter account has a real shell because the synchronization services log in with it over SSH from the other node; see High availability and key synchronization.

On the mailbox server the home of vmail is the mailbox store, created and labelled before Dovecot was started.

$ mkdir -p /data/vmail
$ chown vmail:vmail /data/vmail
$ chcon -R -t mail_home_rw_t /data/vmail/

Two leftovers show in the archived files. /etc/passwd of node A of the pair still has apache and mysql, from the time this machine was the single internal server with webmail; node B has neither. And /etc/group of the mailbox server has a bash-postfix-encrypt-filter group although its install notes never create the account and the filter does not run there.

SELinux

SELinux runs in enforcing mode with the targeted policy. The final design states that for the mailbox server only and the two documents of the first concept state it for their servers; I take it as the setting of all five. The relays needed nothing beyond the stock policy. The internal servers needed one boolean, one file context rule and three local modules.

Each module was built the same way: the denials from the audit log turned into a .te file, the file kept in /etc/selinux, then compiled and loaded by hand. The install notes record the last three steps, here for the Postfix module, as root.

$ cd /etc/selinux
$ checkmodule -M -m -o postfix-local.mod postfix-local.te
$ semodule_package -o postfix-local.pp -m postfix-local.mod
$ semodule -i postfix-local.pp

The notes do not show how the rules were collected. The files carry the section headers that audit2allow writes, and dovecot-local-1.1.te also its #!!!! remarks, so that is where they came from.

Postfix and the filter. Before the module, the log directory of the filter was given the label of /var/log, so that logrotate treats it like any other log. This ran on the internal pair.

$ semanage fcontext -a -e /var/log /var/spool/postfix/bash-postfix-encrypt-filter/log
$ restorecon -R -v /var/spool/postfix/bash-postfix-encrypt-filter/log/

The module of the first concept, printed in the design of 2018, did it differently: it allowed postfix_pipe_t to create and rename files of type var_t, and audit2allow put a warning above that rule that var_t is a base type. Relabelling the directory and allowing var_log_t instead is the cleaner of the two, and it is what the servers ended up with for the file rule. The directory rule for var_t is still in the archived postfix-local.te. The same module was loaded on the mailbox server too, where no filter runs.

Dovecot. A new file system mounted at /data gets the type default_t, which no confined daemon may write. The module allows dovecot_t to do so. On the mailbox server the install notes also relabel the store with chcon to mail_home_rw_t, a type Dovecot may write under the stock policy, so two fixes for the same problem are in place and the notes do not say which one came first. A rule for nfs_t directories and the boolean use_nfs_home_dirs, which the 2018 design lists as enabled together with httpd_unified and httpd_can_network_connect, hint at an early attempt to keep mail on NFS; nothing else in the Source material mentions one. The install notes and the design name the module dovecot-local version 1.0; the archived servers have dovecot-local-1.1 with one more rule, and no note of when it was added.

Keepalived. The module is long because of how killall works: it inspects the executable of every running process to find the ones called master and dovecot, and every daemon on the server became one getattr denial. The list is a census of node A at the time, including httpd and mysqld from its earlier life, and it was copied to node B unchanged. Only two rules are the check itself, the signull permission on the Postfix master and on Dovecot. Three more allow Keepalived to execute systemctl and to start and query units, which nothing in the archived keepalived.conf does. The Source material does not explain them; a rule that nothing needs should not have stayed in the module.

Apache. One boolean on the mailbox server, so that Roundcube can open its connections to Dovecot and Postfix on the loopback address.

$ setsebool -P httpd_can_network_connect on

What I would do differently

• Label, do not allow. chcon on /data/vmail is lost at the next full relabel. A semanage fcontext rule for /data/vmail with mail_home_rw_t, as was done for the filter's log directory, survives it and makes the default_t rules of the Dovecot module unnecessary.
• Read a generated module before loading it. The Keepalived module should have been two signull rules and a dontaudit for the getattr noise, or the check should have been a script that asks for the two process IDs instead of scanning all of them.
• One template for both nodes of a pair. The mount options and even the file system types differ between node A and node B because one was built in 2017 and the other in 2019. Forming the pair was the moment to rebuild node A from the newer template.
• Write down the whole build. The data disk, the number of the filter account and the second version of the Dovecot module exist on the servers and nowhere in the notes.