William Lieurance's Tech Blog

Which TLS library does this use?

2020-10-22 | devops linux

In Debian 10 userspaces, most SSL connections are made via calls to OpenSSL, but LDAPS connections are made with GnuTLS. In RedHat userspaces, LDAPS is still over OpenSSL.

An example: Apache Airflow installed in a Debian 10 userspace is having trouble connecting to an SSL-enabled openldap server speaking LDAPS. You've traced down the problem to a failure to connect, but which tool do you use to validate the connection? You can try to figure out what implementation you're dealing with based on the wording of error messages, but frequently those are wrapped and hidden.

Step one is to figure out how the SSL client is being called. In this example, Airflow is a python app using a framework (Flask-AppBuilder) that is calling python-ldap to construct its client. Python-ldap then, according to its documentation, "wraps OpenLDAP’s client library, libldap." Great, we've figured out where to look next.

Next up is to find how libldap knows how to speak SSL. Since we know it's a compiled library, and almost certainly doesn't implement the SSL protocol itself, it's got to be linking to a different library that does the implementation. The tool to use to query what libraries talk to each other is ldd.

This is where things get a little strange. On the Debian 10 instance in question, the results look like this

$ ldd /usr/lib/x86_64-linux-gnu/libldap-2.4.so.2
        linux-vdso.so.1 (0x00007fffd19f4000)
        liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007fadc379f000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fadc3785000)
        libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fadc3768000)
        libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fadc35bb000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fadc359a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fadc33d9000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fadc33d2000)
        libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fadc32a3000)
        libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fadc3284000)
        libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fadc3100000)
        libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fadc2eed000)
        libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007fadc2eb5000)
        libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007fadc2e7a000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fadc2df7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fadc380a000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fadc2ded000)

That's a list of all of the libraries that libldap dynamically loads. Some of them are standard ones that just about all binaries link to (libc, ld-linux, libpthread), some of them are specific to network connections (libresolv, libsasl2), some low-level crypto libraries (libnettle, libhogweed) and the one that implements SSL is libgnutls.

$ dpkg-query -S /usr/lib/x86_64-linux-gnu/libgnutls.so.30
libgnutls30:amd64: /usr/lib/x86_64-linux-gnu/libgnutls.so.30

No surprise there, GnuTLS owns libgnutls. That means that for this system, we'll need to use the GnuTLS toolchain to do our debugging, including the gnutls-cli command.

Interestingly, compare that to the same library on a RedHat based system.

$ ldd /usr/lib64/libldap-2.4.so.2
        linux-vdso.so.1 (0x00007fffa293b000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f3f52092000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3f52078000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f3f52058000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f3f51fc1000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f3f51cd4000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3f51b0a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3f51b01000)
        libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f3f51ac6000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f3f51a6f000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f3f51984000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f3f5196b000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f3f51964000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f51940000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f3f51926000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3f5212b000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f3f51914000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f3f5190d000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3f518e0000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f3f51845000)

Lots of the same libraries in there, but not exactly. We've got some different low-level crypto library (libcrypt) as well as a different SSL implementer (libssl).

$ rpm -qf /lib64/libssl.so.1.1
openssl-libs-1.1.1g-1.fc32.x86_64

Asking RPM about libssl tells us that it's an OpenSSL toolchain thing, meaning we'll need to use the openssl command to do debugging.

Other Posts