Email 03 - Network, DNS and firewall

Email Solution · Previous: Logical design · Next: Virtual machines and OS build

A mail system is judged from outside by three things it does not control alone: the address its mail comes from, what DNS says about that address, and what the firewalls let through. This part has the two VLANs, every address, the public DNS records, the firewall flows and the routing of the relays, which stand in both VLANs. It is also the part where the design contradicts itself most, so the tables of the design were checked against the configuration of the five site 2 servers, and the result of that check is at the end.

Two VLANs

I read both VLANs as stretched over datacenters A and B of a site: the design does not say so, but a virtual address could not move between the two nodes of the pair otherwise. In site 2 the gateway of the internal network is 10.12.19.46 and 2001:db8:a2:b6f::1, that of the relay network 10.12.19.78 and 2001:db8:a2:b6b::1. Everything that leaves VLAN 1172 passes a firewall that translates the relays' addresses to public ones.

The split repeats on the network level what the concept does with servers. The three internal servers have one interface, in VLAN 1168, and their default route points into the management infrastructure. Only the relays have a second interface, and only that interface has a way to the internet.

Site 2

flowchart TB
  inet["Internet"]
  fw["Firewall, NAT"]
  n1["10.12.19.65 is translated to 203.0.113.44"]
  n2["10.12.19.66 is translated to 203.0.113.48"]
  v1172["VLAN 1172, relay, 10.12.19.64/28"]
  v1168["VLAN 1168, internal, 10.12.19.32/28"]
  vip(["VIP on eth0, 10.12.19.33 and 2001:db8:a2:b6f::f:1"])
  subgraph dca["Datacenter A, hypervisors DC2-A-CKVM001 to 004"]
    ra["DC2-A-VCMSR001"]
    x2["DC2-A-VCMSX002"]
    xa["DC2-A-VCMSX001"]
  end
  subgraph dcb["Datacenter B, hypervisors DC2-B-CKVM001 to 004"]
    rb["DC2-B-VCMSR001"]
    xb["DC2-B-VCMSX001"]
  end
  inet --- fw
  fw --- v1172
  n1 -.- fw
  n2 -.- fw
  v1172 -- "eth1 10.12.19.65, default gateway" --- ra
  v1172 -- "eth1 10.12.19.66, default gateway" --- rb
  v1168 -- "eth0 10.12.19.34" --- ra
  v1168 -- "eth0 10.12.19.35" --- rb
  v1168 -- "eth0 10.12.19.43" --- x2
  v1168 -- "eth0 10.12.19.41" --- xa
  v1168 -- "eth0 10.12.19.42" --- xb
  vip -.- xa
  vip -.- xb

The addresses below are those of the design, and each IPv4 address was found in the ifcfg-eth* file of the server it belongs to.

A relay has two names: vcmsr for the interface the internal servers talk to and vcmsn for the one that is translated.

Site 1

The network figure of site 1 is the same picture with other labels: the same two VLAN numbers, the same five servers with DC1- names on hypervisors DC1-A-CKVM001 to 004 and DC1-B-CKVM001 to 004, the same last octets. A second diagram would add nothing; these are the values.

These are planned values. In October 2019 the new concept was not yet deployed in site 1, and there is no site 1 configuration in the Source material to check them against.

NAT and public DNS

The firewall translates the eth1 address of each relay to one public IPv4 address. The design describes NAT for IPv4 only and says nothing about how, or whether, the relays reach the internet over IPv6.

For a receiving mail server to accept what the relays send, the public DNS of example.net needs four kinds of records per site. This is the set the design specifies for site 1; site 2 has the same set with the domain ad-dc2.example.net and the two 203.0.113 addresses.

The MX records are there for SPF, not for receiving. The SPF policy is written as mx, so the domain must publish MX records that resolve to the relays' public addresses, but the 2017 relay design states that connections from the internet to the relay "will not be allowed", and the firewall tables of 2019 have no flow from the internet inwards. A mail server on the internet that tries to deliver to one of these addresses cannot connect. The Source material holds the specification of the records, not a zone file or a query against the public DNS, so I cannot show what was actually published.

Firewall flows

The design has four tables of six columns, one per server role and site, that repeat the same flows with other addresses. Condensed to one table per role, with the site 2 values, they are these. The source port is 1024-65535 throughout.

The relays have the same nine outgoing flows to the infrastructure services, from 10.12.19.34 and 10.12.19.35, the same SSH flow from the admin VPN, and one flow that nobody else has.

Site 1 has the same flows with its own addresses: the servers in 10.11.19.x, four domain controllers instead of two (10.11.88.145, 10.11.88.146, 10.11.16.209, 10.11.16.210), and the infrastructure services at the same last octets in 10.11.16.x to 10.11.18.x. Infoblox and the admin VPN have the same addresses in the tables of both sites: the Infoblox pair exists in site 1 only, and the VPN network given for site 2 is the site 1 one.

Four remarks on these tables. The client row was never finished: "Any ???" stands in the released version 1.0 of both sites, so the design does not say which networks may reach the mail service. The outgoing rows of the internal table name only the two nodes of the pair as source, not the third server, which needs the directory, DNS and the rest just as much. NTP is listed as TCP; it is UDP 123. And the tables are IPv4 only, although every server has an IPv6 address and the VIP has one too.

The traffic between the mail servers themselves stays inside VLAN 1168 and does not cross the network firewall, so it is in none of the tables. From the configuration it is this.

Each server also runs firewalld. The install notes open the following, as root, on the pair.

$ firewall-cmd --permanent --add-service=ssh
$ firewall-cmd --permanent --add-service=smtp
$ firewall-cmd --permanent --add-port=465/tcp
$ firewall-cmd --reload

Two things do not fit. The design says IMAP is reachable from localhost only, while the notes of the third server open the imap service; the notes contain the firewall block twice, once without and once with it, and do not say which is the final state. And the notes of the pair have no rule for VRRP, IP protocol 112, which a default firewalld zone does not accept from the other node; how the Keepalived advertisements were let through is not recorded.

Routing on the relays

A host with two interfaces has to decide where the default route goes. On the relays it goes out of eth1, towards the internet, because the destinations there cannot be enumerated; the destinations inside can. From ifcfg-eth0 and ifcfg-eth1 of DC2-A-VCMSR001:

# ifcfg-eth0
IPADDR="10.12.19.34"
NETMASK="255.255.255.240"
GATEWAY="10.12.19.46"
DEFROUTE="no"
IPV6_DEFAULTGW="2001:db8:a2:b6f::1"
IPV6_DEFROUTE="no"
# ifcfg-eth1
IPADDR="10.12.19.65"
NETMASK="255.255.255.240"
GATEWAY="10.12.19.78"
DEFROUTE="yes"
IPV6_DEFAULTGW="2001:db8:a2:b6b::1"
IPV6_DEFROUTE="yes"

Both files carry NM_CONTROLLED="no": the interfaces are brought up by the classic network scripts, which read two more files for static routes. route-eth0 has three lines, all through the internal gateway 10.12.19.46: 192.168.0.0/16, 10.0.0.0/8 and 172.16.0.0/12. They are the three private ranges of RFC 1918; together they say "everything private goes inside". The management infrastructure, the directory, DNS, monitoring and the internal mail servers on other subnets are all reached this way, and only what is left over, public addresses, follows the default route to the NAT firewall. The file is a Config document: route-eth0, relay servers.

IPv6 has no private ranges to summarize, so route6-eth0 is a list of twelve /64 prefixes through 2001:db8:a2:b6f::1, one per management network the relay talks to. The prefixes of the Sensu servers (bb1), RabbitMQ (baf) and Graphite (b85) can be recognized in it. The file is identical on both relays and has three flaws as archived: one prefix is listed twice, one line (2001:db8:a2:b99::/64) has no via, and the list is not complete. The prefix of the central syslog server, 2001:db8:a2:b9b::/64, is not in it, and neither is that of the first IPv6 resolver in resolv.conf, 2001:db8:a1:c0b::/64; the list has 2001:db8:a2:c0b::/64 instead. On the first relay whatever is missing leaves through eth1, which has the IPv6 default route; for the second relay, whose eth1 is misconfigured for IPv6, the resulting route is not recorded. A list like this has to be maintained by hand for every new network, which the three IPv4 lines do not. The file is a Config document: route6-eth0, relay servers.

Postfix on the relay needs no setting for any of this. It listens on all interfaces (inet_interfaces = all) and lets the kernel choose the outgoing one; what keeps the internet-facing interface from being a way in is the firewall in front of VLAN 1172 and mynetworks, which holds the VIP and the two nodes of the pair and nothing else. The rest of the relay is in Relay servers and in the Config document Postfix main.cf, relay servers.

What the check against the configuration found

The fourth row is the one with consequences. The relays of site 2 introduce themselves as mx3.ad.example.net and mx4.ad.example.net, hosts of the site 1 domain, and that is myhostname in both main.cf files, so it is what they say in HELO. The design's record tables instead plan mx1 and mx2 under ad-dc2.example.net. Receiving servers compare the HELO name, the PTR record of the connecting address and the SPF record of the sender's domain, and the two naming schemes cannot both be consistent with one set of records. Which records were published for site 2 I cannot tell from the Source material. The mail domain of site 2 itself is not in doubt: the internal servers use ad-dc2.example.net, with one exception on node B that is described in Internal servers: Postfix.

The configuration trees have mistakes of their own that the design does not have.

With the wrong netmask a server treats all of 10.12.19.0/24 as directly connected, which includes the relay network 10.12.19.64/28; the addresses it actually talks to are either inside its own /28 or outside the /24, so the mistake stays without effect as long as nothing internal needs to reach the relays' outer interfaces. The second relay, as archived, has no usable IPv6 address on its outer interface. Its ifcfg-eth1 sets IPV6_DEFAULTGW to the gateway of the relay network and IPV6_DEFROUTE="no"; which IPv6 default route that produced under the classic network scripts is not recorded. Since the design treats the way to the internet as IPv4 with NAT, it is quite possible that nobody noticed; the Source material has no test of outgoing IPv6.