Vault - haproxy.cfg (load balancer)
Vault Solution · Config document · referenced from Load balancer
The HAProxy configuration of both load-balancer nodes of the PROD cluster. HAProxy works in TCP mode and does not terminate TLS: it finds the one Vault node that answers the health check with HTTP 200, which is the active node, and passes the encrypted connection through to it.
| Item | Value |
|---|---|
| Path on the host | /etc/haproxy/haproxy.cfg |
| Deployed on | prod-vault-lb1 and prod-vault-lb2, identical on both |
| Software | HAProxy 1.8 from the Oracle Linux 8 AppStream repository |
| Listens on | the virtual address 10.10.2.14:443 |
| Applied with | systemctl reload haproxy |
The file
#------------------------------------------------------------------------------ # File: haproxy.cfg # Description: haproxy configuration # Author: admin01 # Date: 2024 # # REF: https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/ #------------------------------------------------------------------------------ #------------------------------------------------------------------------------ # Global settings # # Parameters in the "global" section are process-wide and often OS-specific. #------------------------------------------------------------------------------ global # Global log server. # Logging to local rsyslog using facility "local2". log 127.0.0.1 local2 # Start haproxy in a chroot jail in directory "/var/lib/haproxy". chroot /var/lib/haproxy # Location of the PID file. pidfile /var/run/haproxy.pid # Per-process connection limit. maxconn 4000 # Change the process user ID (UID) to dedicated user "haproxy". user haproxy # Change the process group ID (GID) to dedicated group "haproxy". group haproxy # Makes the process fork into background = running as daemon. daemon # Turn on stats unix socket. stats socket /var/lib/haproxy/stats # Utilize system-wide crypto-policies. ssl-default-bind-ciphers PROFILE=SYSTEM ssl-default-server-ciphers PROFILE=SYSTEM #------------------------------------------------------------------------------ # Defaults settings # # Common defaults that all the 'listen' and 'backend' sections will use if not # designated in their block #------------------------------------------------------------------------------ defaults # Using logging configuration from global settings. log global # Defaults mode of operation of instances is TCP mode # The instance will work in pure TCP mode. A full-duplex connection will be # established between clients and servers, and no layer 7 examination will # be performed. mode tcp # Enable of advanced logging of TCP connections with session state and # timers. option tcplog # Enable early logging. # The log message is created as soon as the server connection is established # in mode tcp, or as soon as the server sends the complete headers in mode # http. option logasap # Disable logging of null connections. # We don't want to produce log messages for loadbalancer checks or other # types of port probes. option dontlognull # Number of retries to perform on a server after a failure. retries 3 # Maximum allowed time to wait for a complete HTTP request. timeout http-request 10s # Maximum time to wait in the queue for a connection slot to be free. timeout queue 1m # Maximum time to wait for a connection attempt to a server to succeed. timeout connect 10s # Maximum inactivity time on the client side. timeout client 1m # Maximum time for pending data staying into output buffer. timeout server 1m # Maximum allowed time to wait for a new HTTP request to appear. timeout http-keep-alive 10s # Additional check timeout, but only after a connection has been already # established. timeout check 10s #------------------------------------------------------------------------------ # Frontend settings - PROD-VAULT-LB #------------------------------------------------------------------------------ frontend prod-vault-lb # Listening IP address and port for this frontend. bind 10.10.2.14:443 # Default backend for this frontend. default_backend prod-vault #------------------------------------------------------------------------------ # Backend settings - PROD-VAULT #------------------------------------------------------------------------------ backend prod-vault # HTTP health check of Vault nodes. # We are balancing traffic only to Vault node which is initialized, unsealed # and active = HTTP response code is 200. option httpchk GET /v1/sys/health # HTTP health check is expecting the HTTP response code 200 to mark Vault # node as Online/UP. http-check expect rstatus 200 # Definition of backend servers. # server <name> <address>[:[port]] [param*] # # check = enables health checks on a server # check-ssl = forcing encryption of all health checks over SSL # verify none = client certificate is not requested # inter 5000 = interval between two consecutive health checks server prod-vault-node1 prod-vault-node1.example.net:8200 check check-ssl verify none inter 5000 server prod-vault-node2 prod-vault-node2.example.net:8200 check check-ssl verify none inter 5000 server prod-vault-node3 prod-vault-node3.example.net:8200 check check-ssl verify none inter 5000 server prod-vault-node4 prod-vault-node4.example.net:8200 check check-ssl verify none inter 5000 server prod-vault-node5 prod-vault-node5.example.net:8200 check check-ssl verify none inter 5000 #------------------------------------------------------------------------------ # NOTES #------------------------------------------------------------------------------ # Vault has a http health endpoint on /v1/sys/health that returns different http codes # depending on the status of the running vault instance. # # REF: https://www.vaultproject.io/api/system/health.html#read-health-information # # * 200 if initialized, unsealed, and active # * 429 if unsealed and standby # * 472 if data recovery mode replication secondary and active # * 473 if performance standby # * 501 if not initialized # * 503 if sealed
Per-environment differences
| Setting | PROD | NONPROD | COMMON |
|---|---|---|---|
frontend name | prod-vault-lb | nonprod-vault-lb | common-vault-lb |
bind | 10.10.2.14:443 | 10.20.2.14:443 | 10.30.2.14:443 |
backend name | prod-vault | nonprod-vault | common-vault |
server lines | five nodes | five nodes | three nodes |
Reading it today
- The health check trusts any certificate.
check-ssl verify noneencrypts the check and verifies nothing. With the internal CA already in the system trust store,verify required ca-file /etc/pki/tls/certs/ca-bundle.crtand acheck-sniwould have cost one line. - Newer Vault health codes. Since Vault 1.19
/v1/sys/healthcan also answer 474 (a standby that cannot reach the active node) and 530 (a node removed from the cluster). Both are not 200, sohttp-check expect rstatus 200already treats them as down and needs no change. See the health endpoint reference. - Vault never sees the client address. In TCP mode every request reaches Vault from the load balancer's address, and that is what the audit log records. The supported fix is the PROXY protocol:
send-proxy-v2on theserverlines andproxy_protocol_behaviorwithproxy_protocol_authorized_addrsin the Vault listener. See the TCP listener reference. option httpchksyntax. HAProxy 2.2 and later preferhttp-check send meth GET uri /v1/sys/health; the old one-line form still works.