NetApp MetroCluster - primary storage for a management infrastructure

Between 2017 and 2019 I designed, built and ran the primary storage of the management infrastructure of a network service provider, twice: once in each of two sites. It held the disks of the virtual machines that ran the infrastructure, the audit trails of the session-recording appliances, and it had to keep serving when a whole datacenter went dark.

This Solution is the whole of that system: in each site a fabric-attached MetroCluster of two FAS8200 HA pairs, with Brocade FC switches, ATTO FC-to-SAS bridges and DS224C shelves; NFS for KVM clusters; volume encryption; administrator logins through Active Directory; syslog, SNMP and AutoSupport; Unified Manager; a Tiebreaker for automatic switchover; one recorded upgrade; and the incidents on the way.

The system in one picture

flowchart TB
  clients["NFS clients: KVM hosts, Balabit SCB"]
  subgraph s1["Site 1, DC1"]
    subgraph a["Datacenter A"]
      ca["Cluster DC1-A-XNAS001, FAS8200 HA pair"]
      fa["2 Brocade 6505, 2 ATTO 7500N, 6 DS224C"]
      ca --- fa
    end
    subgraph b["Datacenter B"]
      cb["Cluster DC1-B-XNAS001, FAS8200 HA pair"]
      fb["2 Brocade 6505, 2 ATTO 7500N, 6 DS224C"]
      cb --- fb
    end
    fa == "two FC fabrics, SyncMirror and NVRAM mirror" === fb
    ca -. "cluster peering, SVM configuration" .- cb
    ocum["Unified Manager"]
  end
  subgraph s2["Site 2, DC2"]
    tb["Tiebreaker for site 1"]
    mc2["The same MetroCluster again"]
  end
  clients -- "NFS" --> ca
  ocum --> ca
  ocum --> cb
  tb -. "watches" .-> ca
  tb -. "watches" .-> cb

The fictional environment

Every article and every Config document uses the same names and addresses.

Site 1 is the worked example throughout. The networks of site 1 that come up most often:

The directory is ad.example.net, NetBIOS name EXAMPLE. Administrators are admin01 to admin06.

Articles

Read in this order; it goes from why, through how it was built, to how it was run.

Configuration

ONTAP, Fabric OS and the Tiebreaker are configured by commands, not by files. A Config document for them holds the command set as it was run. Where the Source material holds only the result and not the commands, the document is a listing of the state as built, and says so.

Fabric and bridges

ONTAP, as built

ONTAP, access and security

ONTAP, logging and monitoring

ONTAP, operations

Management software on RHEL 7

Hardware and software

What I would do differently

The articles say this where it belongs. The short list:

• Keep what the device says, not only what was sent to it. The archived switch configuration files and the cabling tables contradict each other, and no switchshow was kept to settle it. See article 3.
• The key manager passphrase and backup need a named home before the first volume is encrypted, and a restore from them was never exercised. See article 8.
• One-year certificates on four clusters ran out once. The first ones expired in November 2018 and their successors are valid from January 2019; the notes do not explain the gap. See article 8.
• The first Active Directory integration was the wrong mechanism. Weeks went into the LDAP name service before the domain tunnel, and the diagnostic commands that would have said so came last. See article 7.
• The weakest device sits in front of every disk. The bridges are managed over HTTP with one local account, and only their network placement protects them. See article 1.
• Hardware-assisted takeover was inactive on all four nodes until somebody read the event log. See article 13.
• Test the unplanned switchover, and write down how the failure was injected. The clean test hid what the dirty one showed. See article 11.
• Half the notes stop before the answer. The installation log, the disk rearrangement and the login investigation all end without their conclusion, and these articles say so instead of inventing one.