HashiCorp Vault - secrets management for a service platform

Between 2021 and 2024 I designed, built and ran the secrets management of a multi-tenant network service platform. Customers of that platform hand over credentials: cloud account keys, VPN pre-shared keys, private keys of certificates, routing passwords. Before Vault, those values travelled through the same data model as everything else. Afterwards a customer portal wrote them into Vault, an orchestration platform read them out when it configured a device, and nothing in between ever held them.

This Solution is the whole of that system: three self-managed Vault clusters on Integrated Storage, one of which exists only to unseal the other two, behind HAProxy and Keepalived, on hardened Oracle Linux 8, with audit logs and snapshots shipped to object storage.

The system in one picture

flowchart LR
  subgraph consumers["API clients"]
    portal["service-portal<br/>writes secrets"]
    orch["orchestrator<br/>reads secrets"]
  end
  subgraph prod["PROD"]
    plb["HAProxy + Keepalived<br/>prod-vault:443"]
    pv["prod-vault<br/>5 nodes, Raft"]
    plb --> pv
  end
  subgraph nonprod["NONPROD"]
    nlb["HAProxy + Keepalived<br/>nonprod-vault:443"]
    nv["nonprod-vault<br/>5 nodes, Raft"]
    nlb --> nv
  end
  subgraph common["COMMON"]
    clb["HAProxy + Keepalived<br/>common-vault:443"]
    cv["common-vault<br/>3 nodes, Raft"]
    clb --> cv
  end
  portal --> plb
  orch --> plb
  portal -. test and dev .-> nlb
  orch -. test and dev .-> nlb
  pv -- "Transit auto-unseal" --> clb
  nv -- "Transit auto-unseal" --> clb
  s3[("S3 object storage<br/>snapshots, audit logs")]
  pv --> s3
  nv --> s3
  cv --> s3

The fictional environment

Every article and every configuration file uses the same names and addresses.

Hosts are named <env>-vault-node1 to node5, <env>-vault-lb1 and lb2, and <env>-vault-bastion1, all under example.net. Shared services (DNS, NTP, package repository, monitoring, log collector, object storage) sit in 10.40.0.0/16, administrators and API clients arrive from 10.41.0.0/16 and above, and addresses that were public in reality are taken from the documentation ranges of RFC 5737.

Articles

Read in this order; it is the order in which the system was built.

Configuration files

Each file is a document of its own: what it is, where it lives, the file with its comments, what differs per node or environment, and what has changed since.

Vault

Vault policies

Load balancer

Linux build and hardening

Logging

Software

What I would do differently

The articles say this where it belongs. The short list:

• The unseal token had a fixed three-year lifetime. It was created in July 2023; a cluster restarted after July 2026 with that token would stay sealed. See article 10.
• The unseal token sat inline in vault.hcl, in a file with mode 0644. See vault.hcl.
• The audit log shows the load balancer as the client of every request, because HAProxy passes TCP through without the PROXY protocol. See article 8.
• The KV v2 policies grant more than they say, because one wildcard stands where data or metadata should. See article 11.
• Two availability zones cannot give a five-node cluster a quorum on their own. The design relied on the platform restarting virtual machines in the other zone. See article 3.