Proxy 06 - SELinux and client trust

Proxy Solution · Previous: Certificate renewal

Two things stood between a configured proxy and a working one, and neither was in Squid. On the datacenter 2 host SELinux stopped the ssl_crtd helper from using the certificate database that had just been created, and Squid would not start. In the lab the first real client, pip on Python 2.7, refused the certificates the proxy generated until it was told where the system keeps its trusted CAs. This Article is about those two problems, the module built from the audit log, the certificate placed in the system anchors, and the one environment variable the note records as the solution for pip. The module file and the two command sets are Config documents: squid-local.te, the lab CA certificate commands and the pip test commands.

The SELinux problem

The datacenter 2 note states it in three lines. The problem: Squid does not start and the log holds the message (ssl_crtd): Uninitialized SSL certificate database directory: /var/lib/ssl_db. To initialize, run "ssl_crtd -c -s /var/lib/ssl_db". The solution: even though the Squid SSL database is initialised correctly, SELinux blocks this function, and the fix is an SELinux policy for Squid, specifically for the ssl_crtd utility. The notes hold neither the audit messages nor the Squid log line in full, only that sentence and the module that came out of it.

Why SELinux objected, as I understand the targeted policy: ssl_crtd -c -s /var/lib/ssl_db creates a new directory under /var/lib, and a directory created there with no file context of its own gets the label of its parent, var_lib_t, the generic type of /var/lib. Squid and the helpers it starts run in the domain squid_t, and the policy of RHEL 7.5 has rules for squid_t on Squid's own types, the cache, the log, the configuration, but none that let it write files of the generic var_lib_t. So the helper is refused when it tries to open and lock the database it was asked to use, and what it reports upwards is not "permission denied" but "uninitialised", because an unreadable database and a missing one look the same to it. That is why my note insists that the database was initialised correctly: the error message points the wrong way, and the audit log is where the real cause was.

flowchart LR
  sq["squid, domain squid_t"] --> h["ssl_crtd helper"]
  h -- "open, lock, write" --> d["/var/lib/ssl_db, label var_lib_t"]
  d -- "denied, audit.log" --> a["audit2allow -M squid-local"]
  a --> r1["allow squid_t var_lib_t dir add_name write"]
  a --> r2["allow squid_t var_lib_t file create getattr lock open read write"]
  r1 --> m["semodule -i squid-local.pp"]
  r2 --> m
  m -- "allowed" --> d

The module from the audit log

The fix was built the way audit2allow is meant to be used: take every denial that names Squid out of the audit log, turn it into a module, load it. As root on dc2-a-vcprx001, in /etc/selinux:

$ cd /etc/selinux
$ grep squid /var/log/audit/audit.log | audit2allow -M squid-local
$ semodule -i /etc/selinux/squid-local.pp
$ cat /etc/selinux/squid-local.te

audit2allow -M writes two files named after its argument, the readable squid-local.te and the compiled squid-local.pp; semodule -i loads the compiled one into the running policy, where it stays across reboots. The .te file is squid-local.te, exactly as the tool printed it. It has one require block and two allow rules:

Between the two rules audit2allow put a line of its own: #!!!! WARNING: 'var_lib_t' is a base type. The warning is the important part of the file. var_lib_t is not Squid's type, it is the type of /var/lib and of everything under it that has no more specific label, so the second rule lets squid_t create and write files anywhere in /var/lib that is labelled with the base type, not just in ssl_db. The module does what the audit log asked for and no more, but what the audit log asked for was wide, because the directory had the wrong label to begin with. A check against the policy package of RHEL 7.5, selinux-policy-targeted-3.13.1-192.el7, shows that the right label was already there: its file contexts carry /var/lib/ssl_db(/.*)? as squid_cache_t, added upstream in March 2017, and squid_t may manage squid_cache_t directories and files. The RHEL 7 squid package does not ship the directory, so when ssl_crtd -c -s /var/lib/ssl_db created it from a root shell it inherited var_lib_t from /var/lib, and nothing relabelled it. The whole fix would have been restorecon -Rv /var/lib/ssl_db; semanage fcontext is only needed for a path the policy does not know. The notes did not look for it; the blunt fix is the one my note, filed under 2018, ends with, and nothing after semodule -i is in it. The Config document keeps the module as it was built.

The note shows this on dc2-a-vcprx001 only. The datacenter 1 note holds no SELinux section and the lab note none either; whether the same module was loaded there, whether SELinux was permissive, or whether the problem never showed, the notes do not say.

The certificate in the system anchors

The lab's last Squid step was to make the host trust the proxy's own certificate. As root on proxy.lab.example.net:

$ cp /etc/pki/tls/certs/proxy.lab.example.net.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust

This is the RHEL mechanism for adding a CA: a certificate in /etc/pki/ca-trust/source/anchors/ is merged by update-ca-trust into the generated bundles, among them /etc/pki/tls/certs/ca-bundle.crt, which is what OpenSSL-based programs on the host read. After it, a certificate for pypi.python.org signed by proxy.lab.example.net chains to a trusted anchor as far as the system bundle is concerned. What this does for the host, the colleague's deployment of the previous Article does for every server in production; the notes hold nothing of that deployment.

The pip test

The test installed pip from EPEL, pointed the shell at the proxy and ran an upgrade through it. The command set is the pip test commands; the part that matters, as root, with http_proxy and https_proxy both set to http://10.90.114.114:3128:

$ pip install -U pip

Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) - skipping
Requirement already up-to-date: pip in /usr/lib/python2.7/site-packages

The proxy bumped the connection and answered with its own certificate for pypi.python.org, and pip refused it, although the proxy host's bundle had been rebuilt with the proxy's certificate in it, if the test ran there. The notes give the solution in one line:

$ export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt

Why that line and not the bundle rebuild alone is, as I understand it, a property of pip rather than of the proxy: pip verifies TLS with the requests library it bundles, and requests carries its own list of CA certificates instead of reading the system's, so an anchor added with update-ca-trust is invisible to it until REQUESTS_CA_BUNDLE points it at the system file. My links below are the pip issues this came out of. The notes hold no second pip install after the fix; that it worked is implied by the title "Solution" and nothing else, and on which machine the test ran, the proxy itself or the lab client at 10.90.114.1, the notes do not say.

flowchart LR
  pip["pip, Python 2.7"] -- "https_proxy" --> px["proxy 10.90.114.114:3128"]
  px -- "certificate for pypi.python.org, signed by the proxy" --> pip
  pip -- "verifies with" --> env["REQUESTS_CA_BUNDLE"]
  env --> b["/etc/pki/tls/certs/ca-bundle.crt"]
  b -- "built by update-ca-trust from" --> an["/etc/pki/ca-trust/source/anchors/"]
  an --> crt["proxy.lab.example.net.crt"]

My links

The lab note ends with the references I kept at the time: the open pip issues about SSL at github.com/pypa/pip, issues with ssl, and three of them in particular, pip issue 5236, pip issue 4919 and pip issue 4459; the Squid wiki page Feature: SslBump Peek and Splice from 2017; and two blog posts on the RHEL trust store, Update and add CA certificates bundle in RedHat and CentOS from 2016 and Adding a new trusted certificate authority from 2015.

What is missing

The audit log lines that audit2allow read are not in the notes, nor the Squid log beyond the one message. There is no test of any client but pip: curl, yum, a browser, anything in Java with its own key store, are not in the notes, and what the production servers do to make their own clients trust the proxy, beyond receiving the certificate from the colleague's deployment, is not either. Whether REQUESTS_CA_BUNDLE was made permanent anywhere, and for which users, the notes do not say.

Today

Checked against the current selinux-policy, Squid 7.7 and pip 26.2.1: the squid_cache_t context for /var/lib/ssl_db is in every current policy branch as it was in RHEL 7.5's, and Squid 4 and later default the database to /var/spool/squid/ssl_db on RHEL builds, which is squid_cache_t through the /var/spool/squid context, so a database created at the default path needs no relabel at all. The error message is unchanged apart from the helper's new name, security_file_certgen: Squid 7.7 still throws "Uninitialized SSL certificate database directory" whenever the helper cannot open the database's index file, so an SELinux denial still reads like a missing database. audit2allow -M and semodule -i are still the tooling. On the client side the REQUESTS_CA_BUNDLE fix is still documented by pip, next to --cert and PIP_CERT, but since pip 24.2 "system certificates are used in addition to certifi" through the truststore package, so on a current pip the anchors step alone would have been enough. The client itself is history: pip 21.0 dropped Python 2 in January 2021, 20.3.4 being the last pip for Python 2.7, and current pip needs Python 3.10 or later.

What I would do differently

Run restorecon -Rv /var/lib/ssl_db and read the audit log against the policy's file contexts before building a module: the warning that audit2allow wrote into the module says why, the module as loaded gives Squid write access to every base-typed file under /var/lib, and the label the policy wanted was there all along.