Email 01 - Requirements and concept

Email Solution · Next: Logical design

Every piece of hardware and software in a management infrastructure wants to send mail: alerts from the monitoring, notifications from applications. This part says what the email system was asked to do, how the first concept of 2017 answered that with two servers, and what the high-availability concept of 2019 turned it into: five servers per site. Everything in the later parts follows from the twelve requirements and one changed sentence in this one.

Why the system exists

The organisation ran its management infrastructure in two sites, site 1 (DC1, production) and site 2 (DC2, pre-production), each with two datacenters A and B. The design opens with the reason in one sentence: the hardware and software components deployed there "have the ability to send email notifications and alerts through the SMTP protocol", and something has to accept that mail. Some of it goes to addresses outside the organisation, and administrators and operators needed a place to read mail addressed to the infrastructure's own domain.

The mail domains are the Active Directory domains: ad.example.net for site 1 and ad-dc2.example.net for site 2.

Functional requirements

The list is the same in the first design of 2017 and in the final one of 2019, except that FR12 names both domains in the later version.

FR2 and FR4 pull in opposite directions, and FR11 adds a third class of sender. How the three were reconciled is the subject of Internal servers: Postfix and Active Directory integration. FR12 is the unusual one: no package in the distribution does it, and it led to an in-house content filter, described in Automatic email encryption.

The relay server had a design document of its own in 2017, with a short list that the final design absorbed without repeating it.

Non-functional requirements, constraints, assumptions

NR2 is the sentence that changed. NR1 explains the software: Postfix, Dovecot, Apache, Roundcube and MariaDB, all from the distribution or from EPEL, and nothing that speaks a protocol of its own. C2 fixed the versions: Postfix 2.10.1, as the paths in main.cf show, and for Dovecot the version RHEL 7 shipped, which the Source material does not record.

The first concept, 2017–2018

The first design was written between June and August 2017 as two documents: one for the internal email server, in seven drafts, and one for the relay email server, in a single draft of August 2017. An eighth version of the first followed in November 2018 with "a small update for configuration of distribution list". Both documents still carry the status "draft", and both say their configuration sections are taken from the real implementation in the production environment.

One server did everything a client could see, and one server stood between it and the internet. The documents do not argue the split, but what it achieves is plain: the host that holds mailboxes, talks to the directory and knows the service account's password has no interface in a network that faces the internet, and the host that has one holds nothing. Availability was left to the virtualization layer, as NR2 said.

What the HA concept of 2019 changed

Version 1.0 of the design is dated 8 October 2019 and describes itself as the "new (HA) email infrastructure concept". The reason it gives is the new NR2 and nothing more: the system is now mission critical. The reasoning behind it is mine, not the design's: with one internal server, every restart of it, for a kernel update or after a mistake in main.cf, is an outage for all senders at once.

Three details of the change are worth knowing before reading on.

The pair is active-passive only for the address. Both nodes run Postfix and Dovecot all the time and accept mail on their own addresses; the cluster resource is the VIP and nothing else. In the words of the design, "from service perspective it is a Active-Active deployment".

The third server is not redundant. Mailboxes and the Roundcube database exist once per site, on DC2-A-VCMSX002 in site 2, and if that server is down the pair queues mail for the own domain until it returns. Sending to the internet does not depend on it. The design does not discuss this as a risk.

In site 1 the VIP took over the address of the old server: 10.11.19.33 and 2001:db8:a1:b6f::f:1 are the address of DC1-A-VCMSX001 in the 2017 document and the address of the VIP in the 2019 one, where the node itself moves to 10.11.19.41. A client configured with the address would not need to be touched. The design does not say this was the intention.

The new concept was built in site 2 first. In October 2019 the design says the HA concept "is not yet deployed" in site 1; every site 1 section about installation reads "will be added after implementation", and the one about configuration says it will be updated once the HA concept is implemented there. The configuration trees and install notes this Solution is written from are those of the five site 2 servers; by the dates of the archive they were saved in June and October 2019. Where the design gives site 1 values, mostly names and addresses, I quote them; site 1 configuration is not invented.

Conceptual design

flowchart TB
  subgraph mua["Email clients, MUA"]
    srv["Server"]
    app["Application"]
    apl["Appliance"]
  end
  adm["Administrators and operators"]
  subgraph net["Site 2, management network"]
    subgraph pair["Active-active pair, Keepalived migrates only the VIP"]
      vip(["Virtual IP"])
      x1["Internal email server 1, DC2-A-VCMSX001, stateful, keys and certificates"]
      x2["Internal email server 2, DC2-B-VCMSX001, stateful, keys and certificates"]
    end
    x3["Internal email server 3, DC2-A-VCMSX002, stateful, mailboxes and webmail database"]
    r1["Relay email server 1, DC2-A-VCMSR001, stateless"]
    r2["Relay email server 2, DC2-B-VCMSR001, stateless"]
  end
  inet["Email servers on the internet"]
  mua -- "SMTP with TLS" --> vip
  mua -- "SMTPS" --> vip
  vip --- x1
  vip -. "failover" .- x2
  x1 <-- "bidirectional file sync" --> x2
  vip -- "SMTP transport, own domain only" --> x3
  vip -- "primary relayhost" --> r1
  vip -. "fallback relayhost" .-> r2
  x3 -- "primary relayhost" --> r1
  x3 -. "fallback relayhost" .-> r2
  adm -- "HTTPS" --> x3
  r1 -- "SMTP, with TLS or without" --> inet
  r2 -- "SMTP, with TLS or without" --> inet

This is the conceptual figure of the design, redrawn. It shows site 2; the design states that the same concept "is/will be applied" in site 1 with DC1- names. In one place the figure and the build disagree: the figure draws the relayhost arrows of the third server straight to the relays, while the configuration of DC2-A-VCMSX002 has the pair as its relayhost (10.12.19.41, fallback 10.12.19.42) and the relays do not list it in mynetworks. Mail written in webmail for a recipient outside the own domain therefore passes the pair, and with it the encryption filter, before it reaches a relay; mail for the own domain is delivered on the third server itself. The install notes of that server say the same as the configuration.

Email clients (MUA). SMTP client software on servers, in applications and on appliances. What they have in common is that they send; what separates them is whether they can do TLS and authentication, see Logical design.

Internal email servers. The first and second accept mail from clients, authenticate and authorize the sender, encrypt, and forward: mail for the site's own domain to the third server, everything else to the relays. The third server delivers into mailboxes and serves them through webmail.

Relay email servers. They accept mail from the internal servers only and deliver it to the mail servers on the internet. This is their one function. The internal servers use the relay in datacenter A and fall back to the one in datacenter B when it does not answer.

Directory server. Active Directory is the repository for users, aliases and mail groups. Every device, server or application that sends mail, except those that cannot authenticate, and every person who needs mail has an account there. Site 1 had four domain controllers, two bare-metal and two virtual; site 2 had two virtual ones. The mail servers talk to them over IPv4 only, "due to problems in LDAP libraries", although everything else in the design is dual-stack.

Certification authority. Every channel between the components, and between them and the directory, is encrypted, and every server has its own certificate signed by the organisation's internal CA. Client certificates were considered and rejected for authentication, because many of the mail clients in the infrastructure do not support them. The details are in TLS and certificates.

Users

The design divides the users into two groups, and the split is by what an account may do, not by who owns it.

In Active Directory this becomes three role groups, one per permission, each made of one authorization group per organisation.

Putting an account into a group is the whole act of provisioning: there is no user database on the mail servers. How Postfix and Dovecot ask the directory is in Active Directory integration, and the day-to-day handling of accounts in Accounts, clients and operations.