Balabit 02 - Logical design and integrations

Balabit SCB Solution · Previous: Requirements and concept · Next: Modes of operation and connections

An SCB on its own is a recorder with nobody to authenticate, nowhere to put its recordings and nobody to tell when something breaks. This part names the systems the site 1 cluster was tied to, what each integration was for, which jump servers the users were sent to, and which network flows the design asked the firewall team to open. The flows are where the design document is least consistent, so the second half of the article is as much a list of gaps as a summary.

The components

The logical figure of the design puts the SCB pair in the middle of the management LAN of site 1, between the two kinds of people who use it and the services it depends on. The SCB, the directory, DNS and NTP and the storage are each drawn as a pair; mail, syslog and monitoring as a single server.

flowchart LR
  admins["SCB Administrators"]
  users["SCB Users"]
  subgraph s1["Site 1 - management LAN"]
    subgraph scb["Shell Control Box"]
      scb1["SCB1"]
      scb2["SCB2"]
    end
    subgraph dns["DNS and NTP Services"]
      dns1["DNS1 NTP1"]
      dns2["DNS2 NTP2"]
    end
    subgraph ad["Active Directory Services"]
      ad1["AD1"]
      ad2["AD2"]
    end
    subgraph nas["Network Attached Storage"]
      nas1["NAS1"]
      nas2["NAS2"]
    end
    subgraph jmp["Jump Servers"]
      j1["JUMP_SERVER_1 ... JUMP_SERVER_X"]
    end
    subgraph mail["Email Services"]
      ms["Internal Email Server"]
    end
    subgraph sys["Central Syslog Services"]
      ss["Syslog Server"]
    end
    subgraph mon["Monitoring Services"]
      sn["Sensu Server"]
    end
  end
  subgraph oob["OOB Jump Servers"]
    o1["JUMP_SERVER_1 ... JUMP_SERVER_X"]
  end
  admins -- "SSH, HTTPS" --> scb
  users -- "SSH, RDP, SCP" --> scb
  scb -- "DNS NTP" --> dns
  scb -- "LDAPS" --> ad
  scb -- "CIFS" --> nas
  scb -- "SSH" --> jmp
  scb -- "RDP" --> oob
  scb -- "SMTP" --> mail
  scb -- "SYSLOG" --> sys
  mon -- "SNMPv3" --> scb

Two labels in the original figure are not what was built. The storage line says CIFS, where the backup and archive policies use NFS (see Backup, archive and retention). And the line to the jump servers in the management LAN says only SSH while the line to the out-of-band jump servers says RDP; in the tables of the same document the management LAN has SSH, RDP and SCP jump servers, and every out-of-band jump server is SSH only. I kept the labels as drawn.

What each integration is for

The host names are in adm.example.net. The Active Directory servers are the ones described in Active Directory integration of the Email Solution, the NetApp side in SVMs and NFS, and Sensu in Logging, monitoring and AutoSupport. For most of them the SCB is a client; Sensu is the one that queries the SCB.

A few choices are worth a sentence each. The directory is used twice, with the same service account scb_svc: once for the people who log in to the SCB's web interface (AAA) and once, as the LDAP server policy AD.EXAMPLE.NET, for the users of the controlled connections; both have nested groups switched on, which is what lets a user's company group sit inside a role group. Both are set to "No certificate is required", so the SCB encrypts the LDAPS session but does not check the domain controller's certificate. The details are in Active Directory and access control.

Syslog goes without TLS because, as the design puts it, the central syslog server did not have encryption implemented, and over IPv4 because the SCB did not support IPv6 for syslog. It goes over TCP because the logging engineer asked for TCP; the analysis expected the central server to forward security events to the SIEM, and that forwarding is not in the Source material. Mail authenticates as scb_mail (the password in the design is the placeholder password_for_scb_mail) and does not check the mail server's certificate either. Monitoring is a poll, not a push: Sensu queries the SCB's SNMPv3 agent as the local SNMP user sensu (SHA1, AES), and the agent answers only 10.11.16.129. The local services, the syslog and mail pages are in Basic settings, logging and monitoring.

The certification authority question

Requirement FR9 says every certificate in the management LAN is signed by the internal CA, and the design's component list makes the SCB the exception. The SCB needs a server certificate for its web interface and a timestamping (TSA) certificate for its audit trails, and both must come from the same CA; the internal CA could not issue TSA certificates. So the SCB's own internal CA signs both, and the organisation's CA signs only the IPMI certificates. That is the rule as designed; the certificate folder shows that the organisation's CA issued a certificate for the cluster's web name as well. Certificates and keys has the whole story.

Jump servers

Every user session ends on a jump server, never directly on a production system. The management LAN of site 1 has five.

A sixth Windows jump server, 10.11.19.177, appears in chapter 7 as the target of partner 2's RDP connection and in the connection summary of 2018 as dc1-a-vcrdp003, a second RDP jump server for partner 3.

The out-of-band jump servers stand in the out-of-band networks of sites 1 to 5, two per site and datacenter: one for partner 1, one for the organisation. The design gives them no names yet (every one is …-tobedefined). Outside site 1 they are known only by an IPv6 /64 per site, datacenter and company, which is why the user connections needed IPv6 at all. The pattern is regular enough to show as a rule rather than a list of twenty rows.

The site 1 rows did not stay IPv4 for long. The connections of chapter 7 list the IPv6 ranges 2001:db8:a1:f94::/64 and 2001:db8:b1:f94::/64 (and the f95 pair for the organisation) for site 1 too, and the connection summary marks every out-of-band IPv4 target "NO" and every IPv6 one "YES". How these targets are reached from the SCB's single user address is the subject of Modes of operation and connections.

The communication matrix

Section 6.7 of the design has four tables of network flows: what the SCB itself needs, how administrators reach it, how users reach it, and what the IPMI modules need. All four are reproduced in the Config document Communication matrix. In short:

The source port is 1024:65535 in every row. The user table carries a note that in the first phase only partner 1's VPN was to be monitored, with the organisation's ready. The admin VPN behind ORG_VPN is 10.11.20.0/25, the network the SCB has a static route to; the partner VPN networks are not given in the table.

The same flows as a picture, with the addresses of chapter 7 where the matrix leaves them out:

flowchart LR
  avpn["Admin VPN 10.11.20.0/25"]
  uvpn["Partner and ORG VPNs"]
  sensu["Sensu 10.11.16.129"]
  ibm["SCB management 10.11.16.81"]
  pro["SCB users 10.11.16.65"]
  bck["SCB backup 10.11.18.225"]
  dns["Infoblox 10.11.16.145, 10.11.18.145"]
  ad["AD 10.11.16.209, 10.11.16.210"]
  mx["Mail 10.11.19.33"]
  sys["Syslog 10.11.17.113"]
  nas["NetApp SVM 10.11.18.236"]
  jmp["Jump servers"]
  avpn -- "TCP 443, 22" --> ibm
  uvpn -- "TCP 22, 222, 3389, 2201 to 2204" --> pro
  sensu -- "UDP 161, not in matrix" --> ibm
  ibm -- "53, 123" --> dns
  ibm -- "TCP 636" --> ad
  ibm -- "TCP 25" --> mx
  ibm -- "TCP 514, not in matrix" --> sys
  bck -- "NFS, not in matrix" --> nas
  pro -- "SSH, RDP, not in matrix" --> jmp

What the matrix leaves out matters as much as what it has. There is no row for syslog to 10.11.17.113, none for Sensu's SNMP queries, none for NFS to the storage, and none for the SCB's own connections to the jump servers, which are the whole point of the box; for the last one I drew the SCB's user address as the source, but the design does not say which interface the server side leaves from. The user flows are IPv4 only, although every connection also listens on 2001:db8:a1:c0e::f:1. And the ports of partners added in 2018 (2205 to 2208 and 2210 to 2213) are not in it, because the matrix was written in 2017. Whether the firewall rules on the network side were complete, I cannot tell; they are not in the Source material.

Site 2

The site 2 cluster has no design document; its exported config.xml of September 2018 shows how it was tied in. It is another cluster nine months later, so I list the differences rather than merging them into the site 1 picture.

Site 2 depended on site 1 for its directory, its names and its time, which makes the backbone between the sites part of its availability. The support bundle of the same day shows the slave node's NTP peers unreachable and the domain join test failing; High availability and Operations, upgrades and troubleshooting go into that.

Loose ends

• Mail server address: section 4.1.4 gives dc1-a-vcmsx001 with 10.11.19.34, the integration table and the matrix 10.11.19.33. In the Email Solution's later plan Network, DNS and firewall, 10.11.19.33 is the virtual address of the internal pair (dc1-s-xcmsx001) and dc1-a-vcmsx001 is one node of it, at 10.11.19.41. Chapter 7 configures the node's name, not the virtual one, so as I read it the SCB's mail depended on one node of the pair. The integration table also gives the password as "Password for scb@ad.example.net", a typo for scb_mail.
• SMTP port: the infrastructure table says TCP 25 with STARTTLS for the SCB, the out-of-band table SMTPS 465 for the IPMI modules, and the IPMI's own SMTP page port 25 (see IPMI out-of-band management).
• The out-of-band table sends LDAPS to "AD server 2" at 10.11.18.210. Everywhere else the second domain controller is 10.11.16.210; 10.11.18.210 is node B's redundant heartbeat address.
• Default route: chapter 7 sets the IPv4 default gateway to 10.11.17.70, which lies in none of the SCB's networks, while the routes to the admin VPN, Sensu, AD and Infoblox go through 10.11.16.86 on the management network. Mail and syslog have no route of their own and would use the default. The site 2 config.xml has the default gateway 10.12.16.70, an address inside the site 2 user network 10.12.16.64/29; by analogy I suspect 10.11.16.70 was meant in site 1, but that is an inference, and if it is right, mail and syslog left from the user address and not from 10.11.16.81 as the matrix says. The routing is in Hardware, cabling and networks.
• The design's text in 3.5 and 4.1.2 says CIFS; its tables and chapter 7 say NFS. The NetApp Solution tells why.