When the LetsEncrypt ROOT certificate expired, I was unable to access it from an older OS, which was a problem

2021.10.07

table of contents

1 Can't connect to some sites via HTTPS since October 1st?
2 What happens
3 Causes and solutions by OS
4 Steps to forcefully install openssl1.0.2 on CentOS6

In Canada, when you order at Starbucks they ask your name (do they ask that in Japan too?), but when I say Seiken they don't understand at all

I'm a timid saint who always thinks to myself that next time I'll use a stylish English name, but when the time comes I'm too embarrassed to say it

Can't connect to some sites via HTTPS since October 1st?

Following the expiration of "DST Root CA X3" (September 30, 2021), LetsEncrypt's root certificate was changed to a cross-root certificate that also uses the certificate authority "ISRG Root X1," which is valid until 2035. However, since the expiration of "DST Root CA X3" on October 1st, SSL certificate errors have been occurring in some environments when accessing sites that use LetsEncrypt certificates

I'm sure there are many people out there who are having the same problem, so I hope this article helps those of you who are struggling

What happens

I get an SSL error when connecting to a site that uses LetsEncrypt (a cross-root certificate of "ISRG Root X1" and "DST Root CA X3")

Causes and solutions by OS

For CentOS7, Amazon Linux, and Amazon Linux2

In most cases, the root certificate installed in the OS is old and the "ISRG Root X1" root certificate does not exist in the OS's root certificate tree. Update the root certificate with the following command

yum -y update ca-certificates

Problems may also occur if the version of openssl is old (1.0.1 or earlier), so in that case please upgrade openssl to the latest version

Common errors in CentOS6

Cross-signed certificate verification requires openssl 1.0.2 or later, but CentOS 6 is no longer supported and only versions up to 1.0.1 are provided

In versions of openssl 1.0.1 and below, the partial chain function for SSL certificates is not implemented, so certificate verification cannot be performed correctly and an SSL communication error occurs

Below are some examples of common error patterns:

wget example

wget https://site using LetsEncrypt certificate

Execution result

--2021-10-04 20:06:18-- https://site using LetsEncrypt certificate/ Resolving site using LetsEncrypt certificate... xx.xx.xx.xx Connecting to csite using LetsEncrypt certificate|xx.xx.xx.xx|:443... connected. ERROR: cannot verify site using LetsEncrypt certificate's certificate, issued by “/C=US/O=Let's Encrypt/CN=R3”: Issued certificate has expired. To connect to site using LetsEncrypt certificate insecurely, use '--no-check-certificate'.

Openssl command example

openssl s_client -connect Site using LetsEncrypt certificate:443 -servername Site using LetsEncrypt certificate < /dev/null

Execution result

CONNECTED(00000003) ～Omitted～ Start Time: 1633378075 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) --- DONE

An example of a php program that communicates using the openssl library (probably libssl.so.10)

Certificate verification fails in programs that use the get_file_contents function as shown below

Sample Code

<?php $url = "https://LetsEncryptの証明書を利用しているサイト/"; $output = file_get_contents($url); var_dump($output);

Execution result

PHP Warning: file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed in "The executed program" on line 3 PHP Warning: file_get_contents(): Failed to enable crypto in "The executed program" on line 3 PHP Warning: file_get_contents(https://site using LetsEncrypt certificate/): failed to open stream: operation failed in "The executed program" on line 3 bool(false)

There is no problem if you use the curl command or the php curl function

This is probably because the cryptographic communication related libraries used by the curl command include not only the openssl-based libssl.so.10 but also the NSS-based libssl3.so

ldd /usr/bin/curl | grep ssl

libssl3.so => /usr/lib64/libssl3.so (0x00007f43fc44b000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f43fb1c5000)

ldd /usr/bin/wget | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f79a04c6000)

ldd /usr/bin/openssl | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f2872a3a000)

rpm -qf /usr/lib64/libssl3.so

nss-3.44.0-7.el6_10.x86_64

rpm -qf /usr/lib64/libssl.so.10

openssl-1.0.1e-58.el6_10.x86_64

To solve the problem

You will need to modify your program to use the curl command or a curl-based library, or use openssl version 1.0.2 or higher

As mentioned above, CentOS 6 does not provide versions higher than 1.0.1, so if it is difficult to modify the program, it is possible to avoid the problem by obtaining an SRPM from the CentOS 7 repository, modifying it for CentOS 6, building it, and installing it

The original source here .

*This is a procedure that has been proven to work on servers that are actually in operation at our company, but please note that we have not been able to confirm the impact and operation of all of OpenSSL's functions

*Fundamentally, we recommend updating your OS

Steps to forcefully install openssl1.0.2 on CentOS6

Create an rpm package from srpm

Obtaining the openssl 1.0.2 srpm package from the CentOS7 repository

cd /usr/local/src wget --no-check-certificate https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm rpm -i openssl-1.0.2k-21.el7_9.src.rpm

*A directory called rpmbuild will be created under the home directory of the user who executed the rpm command and the source code will be extracted, so move into that directory

cd ~/rpmbuild

Modifying the SPEC file

*"%patch68" is unnecessary processing, so comment it out

sed -i -e "s/%patch68/#%patch68/g" SPECS/openssl.spec

Modifying the patch file

*The glibc function secure_getenv does not exist in CentOS6, so replace it with getenv

sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-env-zlib.patch sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2j-deprecate-algos.patch sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-fips-ctor.patch

Build rpm package

*If the tools required for building are not installed, install them before running rpmbuild

yum install rpm-build krb5-devel zkib-devel lksctp-tools-devel zlib-devel gcc rpmbuild -ba SPECS/openssl.spec

Install the rpm package

Install the required dependency packages (perl-WWW-Curl)

yum install perl-WWW-Curl

Go to the directory where the rpm package was built and install it with the rpm command

cd RPMS/ rpm -Uvh openssl-*

Check if the version has been updated

openssl version

*If it is 1.0.2k, it was successful

Disabling the DST Root CA X3 Certificate

*If the DST Root CA X3 certificate remains in the OS's root certificates, the problem will not be resolved. In any case, it is an expired certificate, so it should be disabled

Update the OS root certificate

yum update ca-certificates

Enable the update-ca-trust function and update the root certificate with the update-ca-trust command once

update-ca-trust enable update-ca-trust

*Immediately after updating the root certificate with yum update ca-certificates, the OS's trusted root certificate file (/pki/tls/certs/ca-bundle.crt) will contain not only the certificate contents but also comments such as the issuer and expiration date

This will not affect the operation of the OS, but if you run the update-ca-trust command without any arguments, the OS's trusted root certificates will be organized into a file containing only the certificate contents and comments. Be
sure to run this command to ensure that only the certificate portion is extracted with the next command.

Extract only the "DST Root CA X3" certificate and save it as a file under /etc/pki/ca-trust/source/blacklist/

perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}'</etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem

*If a file named /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem is created and its contents are as follows, everything is fine. If this does not work, you can create the file manually

#DST Root CA MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw 7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69 ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5 JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ -----END CERTIFICATE----

Disable "DST Root CA X3" certificate

update-ca-trust extract

grep "DST Root CA X3" /etc/pki/tls/certs/ca-bundle.crt

*If nothing is displayed, it's OK

grep "ISRG Root X1" /etc/pki/tls/certs/ca-bundle.crt

*If it shows # ISRG Root X1, it's OK

After updating, try running the program that was giving you the error to see if the problem has been resolved

If a service that provides SSL (such as Apache) is running on the server, we recommend restarting the service

If you found this article useful, please click [Like]!

The person who wrote this article

About the author

Seiken Sato

My sixth sense is sharp when I'm in a pinch, but I'm basically an old-fashioned engineer.I'm
currently working in the Canadian office.

beyond audio effects ? 6 key points