Balabit 09 - Certificates and keys

Balabit SCB Solution · Previous: Active Directory and access control · Next: Audit trails: encryption and replay

The SCB carries more certificates and keys than any other box in this environment: a certificate authority of its own with a web server and a time-stamping certificate, the web certificates of its two IPMI modules, a key pair for encrypting and one for signing audit trails, one more encryption key per auditor, and a GPG key for its configuration backup. This part tells where each of them came from, why the SCB ended up with its own CA against requirement FR9 of the design, and goes through the rounds of certificate requests in my notes, most of which led nowhere.

Why the SCB has its own CA

The design started from the rule of the house. Requirement FR9 says that all certificates will be signed by the internal certification authority, and the pre-design analysis had confirmed that there was one: an organisation CA with a CRL, issuing certificates valid for 365 days. Client certificates for logging in to the web interface were rejected in the analysis: too few SCB administrators to make it worth it.

The SCB, however, needs a certificate that most CAs are not set up to issue. Besides the server certificate of its web interface it uses a time-stamping authority (TSA) certificate, whose key timestamps the audit trails. Constraint C3 of the design records the outcome: the organisation's CA does not support issuing TSA certificates, so only the IPMI certificates are signed by it, and all other SCB certificates by the SCB's local CA. Chapter 4.1.7 adds the second half of the reason: the server and the TSA certificate the SCB uses must be issued by the same CA. The design states this as a property of the product and gives no explanation, and I did not keep one.

So the organisation's CA signed what lives outside the SCB's firmware, the IPMI web interfaces, and the SCB's own CA signed everything the appliance itself presents.

The SCB's own certificates

The SCB generated its CA, server and TSA certificates itself, from the subject fields of chapter 4.10.1 of the design. All three are RSA 2048, signed with SHA-256, and valid for twenty years.

The whole page as the design records it is a Config document: SSL certificate (site 1). Every certificate in my folder is in the certificate inventory.

The site 1 certificates carry a name that the documents later stopped using. In the design's DNS records dc1-s-xblm001.adm.example.net is the cluster's management address 10.11.16.81 and dc1-s-xblb001.adm.example.net its production address; the later network sheets give dc1-s-xblb001 to the management address and call the production one dc1-s-xblb001pro. The SCB made its CA in August 2017 under the old name, which stays in the certificates for twenty years. Chapter 7 of the design prints dc1-s-xblb001 in the same three subjects; whether the certificates were made again or the table was typed with the new name, the documents do not tell. Site 2 got the new naming from the start.

The server certificate also lacks the names people were told to use. Its alternative names are the address and the old name; the URL in chapter 8 of the design, https://dc1-s-xblb001.adm.example.net/, and scb.example.net are not in it, so a browser would report a name mismatch on top of an unknown CA (general browser behaviour). The operation how-to maps all three names to 10.11.16.81 in the workstation's hosts file, see workstation hosts file, and has a section "Import CA certificates to PC" whose steps read "TODO". I read that as: the administrators accepted a certificate warning.

The organisation CA's certificates

The design records one web certificate per IPMI module, signed by the organisation's CA, uploaded with its private key, valid from 19 July 2017 to 20 July 2018. The certificates in my folder are newer: issued on 8 January 2018 for dc1-a-ablb001m.mgmt.example.net and dc1-b-ablb001m.mgmt.example.net, RSA 2048, valid one year. The documents do not say why they were renewed half a year early. The IPMI side is in IPMI out-of-band management.

The folder also holds an organisation-CA certificate for the cluster itself: dc1-s-xblm001.adm.example.net with the alternative names dc1-s-xblb001.adm.example.net and scb.example.net, RSA 4096, for client and server authentication, issued on 19 July 2017, the same day as the first IPMI certificates. Three weeks later the SCB generated its own CA. My reading is that the web certificate was requested from the organisation's CA first and that the TSA problem turned the plan around afterwards; the documents do not say whether this certificate was ever installed, and the design's SSL page shows the internal CA's certificates. It expired in July 2018.

Beside these lies the organisation's root certificate, Internal Root CA 1, valid 2007 to 2027 and signed with SHA-1. The certificate of the issuing Internal CA is not in the folder.

flowchart TB
  subgraph scbca["SCB internal CA, one per site"]
    ca1["CA dc1-s-xblm001 root CA, 2017 to 2037"]
    srv["Server certificate of the web interface"]
    tsa["TSA certificate, timestamps audit trails"]
  end
  subgraph orgca["Organisation CA"]
    root["Internal Root CA 1"]
    int["Internal CA, not in the folder"]
    web["Cluster web certificate dc1-s-xblm001, 2017 to 2018"]
    ipa["IPMI dc1-a-ablb001m, 2018 to 2019"]
    ipb["IPMI dc1-b-ablb001m, 2018 to 2019"]
  end
  subgraph xca["XCA database, self-signed, ten years"]
    enc["SCB-AUDIT-ENCRYPT"]
    sign["SCB-AUDIT-SIGN"]
    aud["SCB-AUDIT-ENCRYPT-adminNN, one per auditor"]
  end
  gpg["GPG key pair SCB-BACKUP"]
  scb["SCB cluster"]
  ca1 --> srv
  ca1 --> tsa
  root -.-> int
  int --> web
  int --> ipa
  int --> ipb
  enc -- "certificate" --> scb
  aud -. "certificates, site 2 export has six groups" .-> scb
  sign -- "certificate and private key" --> scb
  gpg -- "public key" --> scb

The dashed edge from the per-auditor certificates is drawn so because how they were used together is not in the material; the site 2 export has six encryption certificate groups, the site 1 design one encryption certificate. The dotted edge from the root is my inference: the issuing CA's own certificate is missing, so the chain from the root to it is not shown by any file.

The certificate requests

My requests were made with OpenSSL as root on a RHEL host (the first round shows the directory /root/balabit), with a copy of the system's openssl.cnf named openssl-balabit.cnf. The final state of that file is a Config document: openssl-balabit.cnf. Its changes switch on request extensions, add the alternative names, and add a [ tsa_cert ] section for a time-stamping request. One change does not belong there: extendedKeyUsage = critical,timeStamping also went into [ usr_cert ], the section OpenSSL uses when it signs a certificate itself. As I read it, it never mattered, because the organisation's CA signed with its own extensions, but the file must not be reused as a CA configuration.

The first attempt sits in the part of my notes headed "BALABIT notes - 4.0.7.a". Its alternative names belong to the earlier variant with two NAT-ed secondary networks, described in Hardware, cabling and networks. Its TSA request asked for nothing but the critical timeStamping usage, exactly what, by constraint C3, the organisation's CA could not issue; whether it was ever submitted is not recorded.

The notes kept their mistakes, and the Config documents keep them too. The first attempt's excerpt writes the addresses as DNS.n lines, but the -text check shows IP Address: entries, which only IP.n lines produce, as in the copy of the file kept in the folder. The per-node server rounds are headed "switch the alternative DNS names on" and show the line commented out, and the out-of-band rounds reuse their file names. All keys were written unencrypted, and the TSA request carries -set_serial 0, which does nothing for a request.

The organisation's CA rewrote the subjects and put the common name into the alternative names. The CA administrator also sent a request template of their own, made out for another server of the management LAN: CSR template from the CA administrator. It creates the key with a public exponent of 3, which I would not copy.

flowchart LR
  a["Copy openssl.cnf to openssl-balabit.cnf"] --> b["Edit req_extensions and alt_names"]
  b --> c["openssl genrsa"]
  c --> d["openssl req -new"]
  d --> e["openssl req -text, check the names"]
  e --> f["Request to the CA administrator"]
  f --> g{"What kind of certificate"}
  g -- "IPMI web interface" --> h["Organisation CA issues it for one year"]
  h --> i["Certificate and key uploaded to the IPMI module"]
  g -- "SCB web name" --> j["Issued, use not recorded"]
  g -- "TSA" --> k["Cannot be issued"]
  k --> l["SCB generates its own CA, server and TSA certificates"]

Keys for the audit trails

The audit trails are encrypted with an X.509 certificate and signed with an X.509 key. Chapter 4.10.3 of the design puts it plainly: the certificates are only a container and delivery mechanism, encryption and decryption use the keys. I made the keys outside the SCB in XCA, in one database file BALABIT-x509.xdb. For site 1 that is SCB-AUDIT-ENCRYPT and SCB-AUDIT-SIGN, made on 18 August 2017, RSA 2048, self-signed with SHA-256 for ten years, and later one encryption key per auditor, SCB-AUDIT-ENCRYPT-admin01 to -admin09 (admin01, admin05, admin07, admin08, admin09). Site 2 has the same set with -DC2 in the names. The list of both sites is a Config document: XCA and GPG keys.

The reason for keeping the encryption keys away from the SCB is, as I understand the purpose of the feature, that the people who run the appliance should not be able to read what it recorded: the SCB gets only the encryption certificate, the private key stays with the auditors. The signing pair is the other way round; the SCB holds the private key and signs with it. The documents do not say why XCA was chosen. How the keys are used is the subject of Audit trails: encryption and replay.

Only one per-auditor certificate is in my folder, admin01's for site 1, from 22 May 2018. My working notes list the site 1 keys without admin09, and my site 2 list heads its last block "Site 1".

The GPG key for the configuration backup

The configuration backup is encrypted with GPG. Chapter 4.10.2 of the design defines the key pair SCB-BACKUP: RSA 2048, e-mail scb_mail@ad.example.net, comment "SCB backup encryption", no expiry. Only the public key went to the SCB, where the system backup is set to encrypt the configuration with it; the site 2 config.xml shows config_encryption enabled with a GPG public key. Restoring a backup needs the private key and its passphrase, and my notes record no restore test. The backup itself is in Backup, archive and retention.

Today

Checked against One Identity Safeguard for Privileged Sessions (SPS) 9.0, the successor of the SCB, the two rules that forced the internal CA are unchanged: server and TSA certificate from the same CA, and a critical Time Stamping usage on the TSA certificate, now also with a key usage of non-repudiation and digital signature only. The vendor now recommends certificates from your own PKI, because those generated on the appliance cannot be revoked. The audit-trail signing certificate must carry the code-signing usage, so the XCA signing certificates would have to be made again. On the OpenSSL side, -nodes is deprecated in favour of -noenc and an exponent of 3 is deprecated. XCA 2.5.0 no longer opens databases from before 2.0.0, and going by the release dates this one was made with XCA 1.x. The details are in the "Checked against" sections of the Config documents.

What I would do differently

I would settle the names before letting the appliance make a CA that lives twenty years: the site 1 certificates still carry the old management name and miss the two names the documents tell people to type. I would also finish the "Import CA certificates to PC" section, so that the SCB's CA reached the administrators' browsers and the auditors' players in one documented way, and give each request round its own file names.