NetApp 04 - Network design

NetApp Solution · Previous: Physical design and cabling · Next: Storage design

Fibre Channel carries the disks and the mirror. Everything else is Ethernet: the clients, the administrators, the peering between the two clusters and the service processors. Each node has two 10 GbE ports for all of it. This article is how eight networks were put on those two ports, which addresses and names were given out, what the firewall had to let through, and which rows of the design document were copy-paste slips.

Networks and VLANs

Only the out-of-band network differs between the datacenters. Every other network is one subnet stretched over both, and it has to be: after a switchover the logical interfaces of an SVM come up in the other datacenter with the same addresses.

The KVM network was not planned this way. The first address sheet put the NFS address of the storage into the hypervisors' own network, VLAN 1006, on one of its last two free addresses, with the note that a separate network for KVM storage was "another question". The question was answered with VLAN 1026, a network that holds only the hypervisors' storage interfaces and one address of the storage, and has no gateway.

The two SVMs added in 2019 for platform B and platform C have their own networks as well. Their addresses are in the final design (10.11.40.30 and 10.11.89.94); their VLAN IDs at site 1 are not in the Source material.

From physical port to SVM

flowchart BT
  subgraph ph["Physical ports of one node"]
    e0a["e0a"]
    e0b["e0b"]
    e0m["e0M"]
    a0a["a0a, LACP over e0g and e0h"]
  end
  subgraph vl["VLAN ports"]
    v1022["a0a-1022"]
    v1024["a0a-1024"]
    v1020["a0a-1020"]
    v1026["a0a-1026"]
    v1028["a0a-1028"]
  end
  subgraph lif["Logical interfaces"]
    cl["clus1, clus2"]
    sp["Service processor"]
    icl["intercluster_lif"]
    nm["Node management"]
    cm["cluster_mgmt"]
    d1["nfs_lif1 of VCVSM001"]
    d2["nfs_lif1 of VCVSM002"]
    d3["cifs_lif1 of VCVSM003"]
  end
  e0a --> cl
  e0b --> cl
  e0m --> sp
  a0a --> v1022 --> icl
  a0a --> v1024
  v1024 --> nm
  v1024 --> cm
  a0a --> v1020 --> d1
  a0a --> v1026 --> d2
  a0a --> v1028 --> d3
  cl --> n1["Twinax to the HA partner"]
  sp --> n2["VLAN 12"]
  icl --> n3["VLAN 1022"]
  nm --> n4["VLAN 1024"]
  cm --> n4
  d1 --> n5["VLAN 1020, Balabit SCB"]
  d2 --> n6["VLAN 1026, KVM hypervisors"]
  d3 --> n7["VLAN 1028, backup"]

The drawing of datacenter B in the design is the same picture with a single data VLAN, 1027, for DC1-S-VCVSM004. The as-built report shows more than the drawings: both clusters have all six VLAN ports on both nodes. Cluster B needs the ports of 1020, 1026 and 1028 for the day it takes over the SVMs of cluster A, and cluster A needs 1027 for the reverse.

The ports of one node, from the as-built report:

Three decisions are visible in that table.

• One interface group for everything. e0g and e0h form a dynamic LACP group, one cable to each Nexus switch. Nothing has an address on a0a itself; every network is a VLAN port on top.
• Node management is in-band. The node-management interfaces sit on a0a-1024, beside the cluster-management interface. The copper port e0M carries only the service processor. Administration therefore depends on the data-centre switches, and the service processor is the way in when they are gone.
• One IPspace per data SVM. Each data SVM got its own IPspace with one broadcast domain and one failover group, both named after the VLAN and both holding the VLAN port of node 1 and node 2. An SVM in its own IPspace has its own routing table, so each can have a default gateway of its own, or none.

The complete state, port by port and interface by interface, is a Config document: ONTAP: network state as built. It is a listing of state and not a command set, because the commands that created the interface groups, VLANs, broadcast domains and interfaces were not kept in the notes.

Logical interfaces

The data interfaces, one per SVM:

Each interface exists twice: once on its own cluster, up, and once on the partner cluster under the -mc copy of the SVM, administratively up and operationally down until a switchover. The home node of each NFS data interface is the node that owns the aggregate with the SVM's data, so client traffic does not cross the cluster interconnect in normal operation.

The default routes show which networks are routed:

The intercluster peering was verified during the installation by pinging every intercluster address of the other cluster from each node. On cluster B, at a time when the node names still ended in M:

$ network ping -node DC1-B-ANAS001M -destination 10.11.10.1
$ cluster peer show -instance

Names

In-band names are in adm.example.net, out-of-band names in mgmt.example.net. Infoblox holds an A record and a PTR record for each.

Cluster and intercluster interfaces have no DNS records. The address sheet also reserved a full second set of addresses in every network for a second MetroCluster that was never built: nodes 003 and 004, cluster 002, switches and bridges 003 and 004. Two of those reserved in-band addresses, .43 and .44, are the ones the two management servers ended up with.

The switch side

Datacenter B uses the same port numbers on its own four switches. The MTU is 9000 on every switch port a node is connected to; the notes list that as the first thing to check when the two clusters disagree about an interface. The Source material has the port list handed to the network team and nothing of the Nexus configuration itself, so the port-channel numbers and how the two Nexus switches present one LACP partner are not recorded here.

What the firewall has to allow

The networks of the storage are behind the firewall of the management infrastructure. The design lists the flows in four groups. Datacenter A is shown; datacenter B is the same with its own addresses.

Between the components of a cluster:

From the administrators, who arrive from the admin VPN 10.11.20.0/25:

To and from the infrastructure services:

"All components" is long: service processors, node and cluster management, both management servers, the data interfaces of the SVMs, the FC switches and both ports of every bridge. A bridge has no use for Active Directory, but the design asked for the same rule set for everything.

The networks around the storage

flowchart LR
  subgraph st["Networks of the storage"]
    oob["Out-of-band management"]
    icl["Intercluster"]
    inb["In-band management, with Unified Manager and API Services"]
    bal["Balabit backup and archive, VCVSM001"]
    kvm["KVM storage, VCVSM002"]
    bck["Backup storage, VCVSM003"]
    adt["AD integration, VCVSM004"]
  end
  ca["Cluster DC1-A-XNAS001"]
  cb["Cluster DC1-B-XNAS001"]
  ca --- oob
  ca --- icl
  ca --- inb
  ca --- bal
  ca --- kvm
  ca --- bck
  ca --- adt
  cb --- oob
  cb --- icl
  cb --- inb
  cb --- bal
  cb --- kvm
  cb --- bck
  cb --- adt
  fw(("Router and firewall"))
  inb --- fw
  bck --- fw
  adt --- fw
  fw --- ad["Active Directory, two servers"]
  fw --- dns["DNS and NTP, Infoblox in A and B"]
  fw --- mail["Mail server"]
  fw --- sys["Central syslog"]
  fw --- mon["Monitoring, Sensu"]

The clients sit directly in their storage networks: the Balabit SCB cluster in VLAN 1020, the hypervisors of both datacenters in VLAN 1026. Nothing is routed between a client and its NFS export.

Slips in the design document

The design was written by copying the tables of datacenter A and editing them. Not every edit was made. The as-built report decides each case.

The Source material does not contain the firewall rules that were actually installed, so it cannot be said which of these slips reached them.

Site 2

Site 2 (DC2) repeats the design with its own addresses. What the Source material holds reliably:

The rest of the site 2 address table in the design cannot be used. Its rows for DC2-S-VCVSM002 to 004, for the FC switches and for the bridges still carry the site 1 addresses, and the two nodes of datacenter B are again named as nodes of A. The intercluster addresses and the VLAN IDs of site 2, apart from 2020, are not in the Source material at all.

Site 2 is also where the rule about identical ports on both clusters was learned in practice. In June 2019 metrocluster check reported a warning for the interface of DC2-S-VCVSM005: cluster B could not find a port with connectivity to that address. The notes for that incident start with a checklist that is the summary of this article: the same MTU and VLANs on every switch port, and the same interfaces, ports, broadcast domains and IPspaces on both clusters. The incident is in Troubleshooting.