[Osaka/Yokohama/Tokushima] Looking for infrastructure/server side engineers!

[Deployed by over 500 companies] AWS construction, operation, maintenance, and monitoring services

[Successor to CentOS] AlmaLinux OS server construction/migration service

[For WordPress only] Cloud server “Web Speed”

[Cheap] Website security automatic diagnosis “Quick Scanner”

[Reservation system development] EDISONE customization development service

[Registration of 100 URLs is 0 yen] Website monitoring service “Appmill”

[Compatible with over 200 countries] Global eSIM “Beyond SIM”

[If you are traveling, business trip, or stationed in China] Chinese SIM service “Choco SIM”

[Global exclusive service] Beyond's MSP in North America and China

[YouTube] Beyond official channel “Biyomaru Channel”

When LetsEncrypt's ROOT certificate expired, I was unable to access it from an old OS.

2021.10.07

In Canada, when I order at Starbucks, they ask me my name (did they ask me that in Japan?) Even though I say Seiken, they don't understand my name at all.

I'm a timid sage who always thinks about using a fancy English name next time, but when the time comes, I'm too embarrassed to say it.

Unable to connect to some sites using HTTPS after October 1st?

Due to the expiration of "DST Root CA After October 1, when "DST Root CA

There seem to be many people who are having the same problem, so I hope this article helps them.

Phenomenon that occurs

An SSL error occurs when connecting to a site that uses LetsEncrypt (a cross-root certificate of "ISRG Root X1" and "DST Root CA X3" in the ROOT certificate)

Causes and solutions by OS

For CentOS7, Amazon Linux, Amazon Linux2

Most likely, the ROOT certificate installed on the OS is old and the "ISRG Root X1" root certificate does not exist in the OS's ROOT certificate tree. Update the ROOT certificate with the following command Sho.

yum -y update ca-certificates

The problem also occurs if the openssl version is old (1.0.1 or lower), so in that case, please also update openssl to the latest version.

CentOS6 and common errors

Verification of cross-root certificates requires openssl 1.0.2 or higher, but support for CentOS6 has ended and only versions up to 1.0.1 are provided.

In openssl1.0.1 and lower versions, the SSL certificate partial chain function is not implemented, so the certificate cannot be verified correctly and an SSL communication error will occur.

Below are some examples of common error patterns.

wget example

Sites using wget https://LetsEncrypt certificates

Execution result

--2021-10-04 20:06:18-- https://Site using LetsEncrypt certificate/ Resolving Site using LetsEncrypt certificate... xx.xx.xx.xx Connecting to cLetsEncrypt's certificate |xx.xx.xx.xx|:443... connected. ERROR: cannot verify LetsEncrypt's certificate, issued by “/C =US/O=Let's Encrypt/CN=R3”: Issued certificate has expired. To connect to sites using LetsEncrypt certificate insecurely, use '--no-check-certificate'.

openssl command example

openssl s_client -connect Sites using LetsEncrypt certificates:443 -servername Sites using LetsEncrypt certificates < /dev/null

Execution result

CONNECTED(00000003) ～Omitted～ Start Time: 1633378075 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) --- DONE

Example of a php program that communicates using the openssl library (probably libssl.so.10)

Certificate validation fails in programs that use the get_file_contents function as shown below.

sample code

<?php $url = "https://LetsEncryptの証明書を利用しているサイト/"; $output = file_get_contents($url); var_dump($output);

Execution result

PHP Warning: file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed in "executed program" on line 3 PHP Warning: file_get_contents(): Failed to enable crypto in "Executed program" on line 3 PHP Warning: file_get_contents(https://sites using LetsEncrypt certificate/): failed to open stream: operation failed in "Executed program" on line 3 bool(false )

The problem does not occur when using the curl command or the php curl function.

I think this is probably because the cryptographic communication related libraries used by the curl command include not only libssl.so.10 of the openssl series but also libssl3.so of the NSS series.

ldd /usr/bin/curl | grep ssl

libssl3.so => /usr/lib64/libssl3.so (0x00007f43fc44b000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f43fb1c5000)

ldd /usr/bin/wget | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f79a04c6000)

ldd /usr/bin/openssl | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f2872a3a000)

rpm -qf /usr/lib64/libssl3.so

nss-3.44.0-7.el6_10.x86_64

rpm -qf /usr/lib64/libssl.so.10

openssl-1.0.1e-58.el6_10.x86_64

In order to solve the problem

You need to modify the program to use the curl command or curl-based library, or use openssl version 1.0.2 or higher.

As mentioned above, CentOS6 does not provide a version higher than 1.0.1, so if it is difficult to modify the program, you can somehow solve the problem by obtaining SRPM from the CentOS7 repository, modifying it for CentOS6, building it, and installing it. It was possible to avoid it.

Here the original story .

*Although this procedure has a proven track record on servers that are actually running at our company, please note that we have not been able to confirm the impact and operation of all openssl functions.

*Basically, we recommend updating the OS.

Steps to forcefully install openssl1.0.2 on CentOS6

Create rpm package from srpm

Obtaining openssl 1.0.2 srpm package from CentOS7 repository

cd /usr/local/src wget --no-check-certificate https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm rpm - i openssl-1.0.2k-21.el7_9.src.rpm

*A directory called rpmbuild is created under the home directory of the user who executed the rpm command, and the source code is extracted there, so move it there.

cd ~/rpmbuild

Modifying the SPEC file

*"%patch68" is an unnecessary process, so comment it out.

sed -i -e "s/%patch68/#%patch68/g" SPECS/openssl.spec

Modifying the patch file

*The glibc function called secure_getenv is not available in CentOS6, so replace it with getenv.

sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-env-zlib.patch sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2j-deprecate -algos.patch sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-fips-ctor.patch

build rpm package

*If the tools required for build are not installed, please install them before rpmbuild.

yum install rpm-build krb5-devel zkib-devel lksctp-tools-devel zlib-devel gcc rpmbuild -ba SPECS/openssl.spec

install rpm package

Install the required package (perl-WWW-Curl) as a dependency

yum install perl-WWW-Curl

Move to the directory where the rpm package was built and install it using the rpm command

cd RPMS/ rpm -Uvh openssl-*

Check if the version is updated properly

openssl version

*If it is 1.0.2k, it is a success.

Disabling DST Root CA X3 certificate

*If the DST Root CA

Update the OS root certificate

yum update ca-certificates

Enable the update-ca-trust function and update the root certificate once with the update-ca-trust command

update-ca-trust enable update-ca-trust

* Immediately after updating the root certificate with yum update ca-certificates, the OS's trusted root certificate (/pki/tls/certs/ca-bundle.crt) file contains not only the certificate content but also the issuer, expiration date, etc. The comment is now included.

There is no problem with the operation of the OS in this state, but if you run the update-ca-trust command without arguments, the OS's trusted root certificate will be organized into a file containing only the certificate contents and comments.
Be sure to run the following command to ensure that only the certificate part is extracted.

Extract only the "DST Root CA X3" certificate and save it as a file under /etc/pki/ca-trust/source/blacklist/

perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_} '</etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem

*It is okay if a file named /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem is created and the contents are as shown below. If that doesn't work, you can create the file manually.

#DST Root CA b290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhv cNAQEBBQADggEPADCCAQoCggEB AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq 5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw 7BZy1SbsOFU5Q9D8/RhcQPGX69 Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD aeQQmxkqtilX4+U9m5/ wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9Lww mglSBd49lZRNI+DT69 ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5 JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo Ob8VZRzI9neWagqNdw vYkQsEjgfbKbYK7p2CNTUQ -----END CERTIFICATE------

Disable "DST Root CA X3" certificate

update-ca-trust extract

grep "DST Root CA X3" /etc/pki/tls/certs/ca-bundle.crt

*It is OK if nothing is displayed.

grep "ISRG Root X1" /etc/pki/tls/certs/ca-bundle.crt

*If # ISRG Root X1 is displayed, it is OK.

After updating, try running the program that caused the error and see if the problem is resolved.

If a service that provides SSL (such as Apache) is running on the server, we recommend restarting the service.

If you found this article helpful , please give it a like!

[2025.6.30 Amazon Linux 2 support ended] Amazon Linux server migration solution

[Osaka/Yokohama] Actively recruiting infrastructure engineers and server side engineers!

The person who wrote this article

About the author

Seiken Sato

I have a good sixth sense when I'm in a pinch, but I'm basically an old-school engineer.Currently, I
work in the Canadian office.

Talking about audio effects What is Beyond's branding? Introduction of 6 points