Proxy - a Squid proxy with SSL interception behind firewalld
In 2018 the servers of a management infrastructure in two datacenters got their way to the internet through one Squid forward proxy per datacenter on Red Hat Enterprise Linux 7.5, with two interfaces, a firewalld zone per interface that drops everything not listed, and SSL interception: Squid holds a CA certificate of its own, looks at the TLS handshake of every connection, lets the excluded servers through untouched and re-signs the rest with a certificate generated on the fly, which the servers trust because the CA was deployed to them. I built it in a lab first, then on the two production machines, and renewed the certificate by hand when it expired. Why the traffic had to be intercepted rather than tunnelled my notes do not say; the only client they show is pip.
This Solution is that work, written up from my working notes: the lab build on a VMware guest, the firewalld configuration of both production proxies with every rich rule as it was entered, the Squid configuration with its SSL-bump steps, the certificate renewal on each host (the notes are filed under 2018 and 2019), the SELinux module that let Squid's certificate generator write its database, and the one client problem that showed what interception costs. The notes are thin in places and say so: the production squid.conf is known only from its two port lines, and nothing of the automation that deployed the CA to the servers is in them.
ssl_crtd helper are long replaced, firewalld's direct rules are deprecated and the ifcfg files of the lab are gone from RHEL 10. The system is anonymized: host names, domain, addresses and the IPv6 prefix are replaced consistently with the Email and NetApp Solutions of the same environment, the certificate request answers are placeholders, and no organisation, place or person is named. The notes contain mistakes and contradict themselves in places; that is reported where it was found, not corrected.The system in one picture
flowchart LR subgraph clients["Clients, zone internal"] srv["Servers in the RFC 1918 ranges and 2001:db8::/32"] ans["Ansible hosts"] sen["Sensu hosts"] end subgraph prx["dc1-a-vcprx001 or dc2-a-vcprx001, RHEL 7.5"] ens3["ens3 internal, target DROP"] squid["Squid 3.5, port 3128, ssl-bump"] ens4["ens4 external, target DROP, no masquerade"] end inet["Internet"] srv -- "TCP 3128, ICMP" --> ens3 ans -- "TCP 22" --> ens3 sen -- "UDP 161" --> ens3 ens3 --> squid squid --> ens4 ens4 --> inet
The fictional environment
Every Article and every Config document uses the same names and addresses.
| Thing | Value |
|---|---|
| Proxies | dc1-a-vcprx001.adm.example.net in datacenter 1, dc2-a-vcprx001.adm.example.net in datacenter 2; RHEL 7.5, Squid from the distribution (3.5.20 at the time, by the package list of RHEL 7.5; the notes print no version), firewalld |
Internal interface ens3 | 10.11.16.113/28 and 2001:db8:a1:b96::f:1 in datacenter 1; 10.12.16.113/28 and 2001:db8:a2:b96::f:1 in datacenter 2; zone internal, proxy port TCP 3128 |
External interface ens4 | 10.11.17.145/28 in datacenter 1, 10.12.17.145/28 in datacenter 2; zone external |
| Clients | the RFC 1918 ranges 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 and the IPv6 2001:db8::/32 |
| Ansible hosts | 10.11.17.241, 10.11.17.242, 2001:db8:a1:b89::f:1, 2001:db8:a1:b89::f:2 (datacenter 1); 10.12.17.241, 10.12.17.242, 2001:db8:a2:b89::f:1, 2001:db8:a2:b89::f:2 (datacenter 2); SSH |
| Sensu hosts | 10.11.16.129 to .131 and 2001:db8:a1:bb1::f:1 to ::f:3 (datacenter 1); 10.12.16.129 to .131 and 2001:db8:a2:bb1::f:1 to ::f:3 (datacenter 2); SNMP |
| Lab | proxy.lab.example.net on VMware, ens32 external 10.90.0.102/24, ens33 internal 10.90.114.114/24, client 10.90.114.1 |
| Certificates | self-signed, /etc/ssl/squid/ in production (365 days), /etc/pki/tls/ in the lab (3652 days); certificate cache /var/lib/ssl_db |
The address plan is the one of the Email and NetApp Solutions; the generic RFC 1918 ranges in the rules are what the notes really used. The vendor parts of the MAC addresses are real, their serial parts are not.
Articles
Read in this order; it is the order the work was done in.
| # | Article | What it covers |
|---|---|---|
| 1 | Overview and design | Why a proxy with interception, one host per datacenter with two interfaces, the lab, what the notes hold and do not hold |
| 2 | Interfaces and firewalld zones | Two ways to bind an interface to a zone, leftover bindings found in production, target DROP, default services and masquerade removed, denied traffic logged |
| 3 | firewalld rich rules | Who may reach the proxy port, SSH and SNMP, ICMP for both address families, the differences between the two datacenters and the rule that pointed at the wrong host |
| 4 | Squid with SSL interception | The default configuration kept, the listening port with its options, the certificate cache, peek, splice, stare and bump, the exclusion lists |
| 5 | Certificate renewal | A self-signed CA renewed yearly: request, signing, the cache rebuilt, the certificate handed on for deployment |
| 6 | SELinux and client trust | Why Squid would not start, the module built from the audit log, the CA in the system anchors, and a pip that refused the proxy's certificates |
Configuration
Each document holds one file, one command set or one listing as it was used, with comments and a check against the current release. The notes show no firewalld file, so its documents are the command sets as they were run, with the output of the checks.
Network and firewalld
Squid
| Document | Explained in |
|---|---|
| squid.conf of the lab | 4 |
| ssl_exclude_domains.conf | 4 |
| ssl_exclude_ips.conf | 4 |
| Listening ports of dc1-a-vcprx001 | 4 |
| Listening ports of dc2-a-vcprx001 | 4 |
Certificates, SELinux and the client test
What the write-up found
Reading the notes again turned up things I did not see when I wrote them. Each is told in its Article.
| Finding | Where |
|---|---|
In the lab, after the interfaces were bound to zones through their ifcfg files, the active zones came out the wrong way round | 2 |
Both production proxies already carried zone bindings to eth0 and eth1, interfaces they did not have, and a source range on the internal zone; datacenter 2 also had the squid service with four open ports, and its source range was never removed | 2 |
external kept the ssh service in the lab and in datacenter 1; only datacenter 2 removed it | 2 |
| On datacenter 2 the IPv6 rule for the proxy port was listed with datacenter 1's address, so IPv6 clients there met a zone that drops, if the listing is right | 3 |
| Datacenter 1 lets only its own Ansible and Sensu hosts in; datacenter 2 lets both datacenters' hosts in | 3 |
| Several step titles name the wrong port, address family or host, copied from the other datacenter's notes | 3 |
| Both production hosts show the same MAC addresses and link-local addresses | 1 |
The production Squid had an http_port 3128 and an https_port 3128 … intercept line, both printed without a comment sign | 4 |
An at_step SslBump3 ACL is defined and never used | 4 |
The certificate cache is owned by squid through a glob that leaves the directory itself to root | 4 |
The SELinux module allows Squid to write to every directory of the base type var_lib_t; the RHEL 7.5 policy already had a label for /var/lib/ssl_db, and restorecon would have been the whole fix | 6 |