Email 02 - Logical design

Email Solution · Previous: Requirements and concept · Next: Network, DNS and firewall

The concept has five boxes per site. This part opens them: which daemon runs in which box, what each one is responsible for, under which account it runs, and which way a message travels from a client to a mailbox or to the internet. It is the map for all the later parts, which each take one daemon and its configuration.

Naming convention

The servers follow the organisation's host naming plan: <Site>-<Node>-<Type><Usage><DeviceID>.

So DC2-B-VCMSX001 is the first internal email server in datacenter B of site 2, and DC2-S-XCMSX001 is a cluster name, stretched over both datacenters: the virtual address of the pair.

Accounts follow a convention too. Every system that sends mail gets its own account in Active Directory, named <service-name>_mail, for example gitlab_mail for the notifications of GitLab. One account per sender means one sender can be switched off, or allowed to send to the internet, without touching another.

Components per site

The design is not consistent about the name of the VIP. Its component tables and its certificate tables write DC2-S-VCMSX001, with the type code of a virtual machine; the Keepalived section, the firewall tables and the client settings write DC2-S-XCMSX001. The install notes settle it: the DNS name of the VIP and the alternative name in the certificates of both nodes is dc2-s-xcmsx001.adm.example.net.

Software and responsibilities

The relays are deliberately the smallest box: Postfix and nothing else, no Dovecot, no directory lookups in Postfix, no content filter. Each of the others has a part of its own: Internal servers: Postfix, Dovecot authentication, Automatic email encryption, High availability and key synchronization, Relay servers, Mailbox server and Webmail.

The logical figure of the design is busy, so it is redrawn here as two. The first shows the servers and the path of mail.

flowchart LR
  mua["Email clients: server, application, appliance"]
  subgraph xa["DC2-A-VCMSX001"]
    vip(["HA VIP, keepalived master"])
    p465["postfix, TCP 465"]
    p25["postfix, TCP 25"]
    mta["postfix MTA"]
  end
  subgraph x2["DC2-A-VCMSX002"]
    m25["postfix MTA, TCP 25"]
    dov["dovecot: AUTH, IMAP"]
    mbox["Mailboxes"]
    web["httpd, TCP 443, webmail"]
  end
  ra["DC2-A-VCMSR001, postfix MTA, TCP 25"]
  rb["DC2-B-VCMSR001, postfix MTA, TCP 25"]
  mua -- "SMTPS" --> vip
  mua -- "SMTP with TLS" --> vip
  vip --> p465
  vip --> p25
  p465 --> mta
  p25 --> mta
  mta -- "SMTP" --> ra
  mta -- "SMTP" --> rb
  mta -- "SMTP" --> m25
  m25 --> dov
  dov --> mbox
  web --> dov

The second shows what is inside the pair and what the two nodes exchange. DC2-B-VCMSX001 runs the same Postfix with the same two ports; only Keepalived differs.

flowchart LR
  ad["Active Directory, DC2-A-VCAD001 and 002, TCP 636"]
  subgraph xa["DC2-A-VCMSX001, on hypervisors DC2-A-CKVM001 to 004"]
    ka["keepalived, master"]
    pa["postfix MTA"]
    da["dovecot AUTH"]
    ga["PGP keys synchronization"]
    sa["S/MIME certificates synchronization"]
    pa <-- "SASL" --> da
  end
  subgraph xb["DC2-B-VCMSX001, on hypervisors DC2-B-CKVM001 to 004"]
    kb["keepalived, backup"]
    pb["postfix MTA"]
    db["dovecot AUTH"]
    gb["PGP keys synchronization"]
    sb["S/MIME certificates synchronization"]
    pb <-- "SASL" --> db
  end
  ka <-- "VRRP, labelled MULTICAST in the figure" --> kb
  ga <-- "SSH, rsync" --> gb
  sa <-- "SSH, rsync" --> sb
  pa -- "LDAPS" --> ad
  da -- "LDAPS" --> ad

The figure labels the link between the two Keepalived daemons "MULTICAST/VRRP". The configuration does not use multicast. Both nodes name each other as unicast peers, and the text of the same design says so in section 4.11 of the same chapter ("cluster members communicate with other members thru unicast").

    unicast_src_ip 10.12.19.41   # IP address of local interface
    unicast_peer {                  # IP address of peer interface
        10.12.19.42
    }

That is node A; node B has the two addresses swapped. The list of links at the end of the install notes holds the reason, a question titled "Keepalived VIP is active on both servers" with the note "Multicast versus Unicast": when the multicast advertisements do not get from one node to the other, each believes it is alone and both take the address. I take it that the figure was never corrected after that; the Source material does not say. The whole file is a Config document: keepalived.conf.

Service accounts

The four kinds of clients

The design sorts the mail clients of the infrastructure by two abilities.

The first row is the rule and what Accounts, clients and operations gives as the client settings. The other three were to be handled as "exceptions for such clients", in the words of FR4, and the design never says what form an exception takes. The configuration of the pair has none: mynetworks holds exactly three addresses, the two relays and the third internal server, and the client restrictions end in reject for everybody who is neither authenticated nor in that list. Whether an appliance without TLS was ever connected, in site 1 or later, the Source material does not show.

SASL layering

Postfix does not check passwords itself. SMTP AUTH is an application of SASL, a framework that separates the protocol that needs authentication from the mechanism that performs it. The design illustrates it with a small diagram, redrawn here.

flowchart TB
  smtp["SMTP"]
  ldap["LDAP"]
  xmpp["XMPP"]
  op["Other protocols"]
  sasl["SASL abstraction layer"]
  ext["EXTERNAL"]
  gss["GSSAPI"]
  plain["PLAIN"]
  om["Other mechanisms"]
  smtp --- sasl
  ldap --- sasl
  xmpp --- sasl
  op --- sasl
  sasl --- ext
  sasl --- gss
  sasl --- plain
  sasl --- om

In this system the SASL implementation is Dovecot, not Cyrus. Postfix hands the exchange to Dovecot over a UNIX-domain socket, private/auth in the Postfix spool, which the design prefers over a TCP socket "for better privacy". The mechanisms are PLAIN and LOGIN, the two plaintext ones, chosen because nearly every mail client supports them; that choice is acceptable only because the password never travels outside TLS. Dovecot then binds to Active Directory over LDAPS with what the client sent. Three Postfix settings complete the picture: noanonymous as the only mechanism property, a sender-to-login map, intended to keep an authenticated client from using somebody else's address, and permit_sasl_authenticated as the way in. The second did not take effect as archived: the order of the sender restrictions defeats it, see Internal servers: Postfix.

Mail flow

sequenceDiagram
  participant c as Client
  participant p as Postfix on the pair
  participant d as Dovecot auth
  participant ad as Active Directory
  participant f as Encrypt filter
  participant m as DC2-A-VCMSX002
  participant r as Relay
  participant i as Internet
  c->>p: Connect to the VIP, port 465 or 25 with STARTTLS
  c->>p: AUTH PLAIN or LOGIN
  p->>d: Check over the socket private/auth
  d->>ad: Bind over LDAPS as the user
  ad-->>d: Result
  d-->>p: Result
  c->>p: MAIL FROM, RCPT TO, DATA
  p->>ad: LDAP lookups for sender allowed, external allowed
  p->>f: One copy per recipient
  f->>p: Encrypted or unmodified, back through sendmail
  alt Recipient in the own domain
    p->>m: SMTP to 10.12.19.43
    m->>m: Delivery by Dovecot into the Maildir
  else Any other recipient
    p->>r: SMTP to the relayhost, or to the fallback relay
    r->>i: SMTP, headers stripped, through NAT
  end

From a client to the internet, in words. The client connects to the VIP and is therefore talking to whichever node holds it; it could equally connect to either node's own address, which is why the design calls the pair active-active. After TLS and authentication Postfix asks the directory whether the sender is in SMTP_ACCESS and, for a recipient outside the own domain, whether the sender is in ESMTP_ACCESS. The design intended a third question, whether the sender address belongs to the login; as archived, a member of SMTP_ACCESS is permitted before that check is reached, so it never takes effect, see Internal servers: Postfix. An accepted message is queued and given to the content filter, one recipient at a time, because the decision to encrypt, and with which key, is per recipient. The filter hands its result back to Postfix. The transport table then decides: the own domain goes to the third internal server, and everything else follows relayhost to DC2-A-VCMSR001, or smtp_fallback_relay to DC2-B-VCMSR001 when the first does not answer. The relay removes the headers that describe the inside (Received, User-Agent, X-Mailer, X-Originating-IP) and delivers to the mail server of the recipient's domain, with TLS when that server offers it.

From a client to a mailbox the first half is the same. The third server accepts the message from the pair because the pair is in its mynetworks, looks the recipient up in the directory, and passes the message to Dovecot's delivery agent, which writes it as the user vmail into /data/vmail/<domain>/<user>/Maildir. The format is Maildir, one file per message, so that no program has to implement locking. The owner of the mailbox reads it in Roundcube, which talks IMAP to the Dovecot on the same host; there is no IMAP client access in the design other than webmail.

There is one more path, in the other direction. The relays have a transport entry of their own that sends mail for ad-dc2.example.net to the third internal server; the install notes explain it as "mainly non delivery email notifications", the bounces a relay generates when the internet refuses a message. The whole configurations are Config documents: Postfix main.cf, internal servers and Postfix main.cf, relay servers; the filter is bash-postfix-encrypt-filter.sh.

Infrastructure services every server uses

The mail servers are ordinary members of the management infrastructure, and each of the five carries the same set of clients for its shared services. This is the second logical figure of the design, with the site 2 names.

flowchart LR
  adm["Administrators and operators"]
  subgraph vm["Any of the five virtual servers"]
    sshd["sshd"]
    sssd["sssd, AD integration"]
    dns["DNS resolver"]
    ntp["ntpd"]
    rsys["rsyslogd"]
    sensu["sensu-client"]
    audit["auditd"]
    etck["etckeeper"]
  end
  ad["Active Directory, DC2-A-VCAD001 and 002"]
  ibx["Infoblox, in site 1"]
  sys["Central syslog, dc2-s-xcsys001"]
  rmq["RabbitMQ, dc2-a-vcrmq001 and dc2-b-vcrmq001"]
  gra["Graphite, dc2-s-xcgra001"]
  sns["Sensu servers"]
  git["GitLab"]
  adm -- "SSH, TCP 22" --> sshd
  sssd -- "AD, LDAP" --> ad
  dns -- "TCP and UDP 53" --> ibx
  ntp -- "NTP 123" --> ibx
  audit --> rsys
  rsys -- "syslog, TCP and UDP 514" --> sys
  sensu -- "TCP 5672" --> rmq
  sensu -- "metrics, TCP 80" --> gra
  sensu -- "ICMP" --> sns
  etck -- "git over SSH, TCP 22" --> git

The Infoblox pair exists in site 1 only. /etc/resolv.conf on all five site 2 servers lists the two site 1 addresses and two IPv6 resolvers, with options timeout:1 attempts:1 rotate, so name resolution and time in site 2 depend on the link to site 1. For a mail server that is not a detail: Postfix resolves the MX of every external recipient.

Monitoring adds a few process checks to the base template every server gets (sshd, sssd, ntpd, qemu-ga, rsyslogd, auditd, crond, file system usage).

Every check is "the process must be running". Nothing in the design sends a test message through the chain or looks at the length of the queue, and nothing checks Keepalived or the two synchronization services. The addresses, VLANs and firewall flows behind all of this are in the next part, Network, DNS and firewall.