Manual Verification of SSL/TLS Certificate Trust Chains using Openssl (Part 1/2)

This week, during my Internet Storm Center (ISC) shift, Firefox 3.6.3 (the latest available version) displayed a digital certificate error when accessing the ISC login page through SSL/TLS: https://isc.sans.org/myisc.html. I confirmed this on a couple of Firefox instances running on Mac OS X and Windows XP.

We also got a few reports from ISC readers on the same issue, although other people running the same browser version, and even language (EN), on the same OS platforms, didn't get any error message. Finally, the reason was a new ISC digital certificate had been recently installed, and the required intermediate certificate was missing in some web browsers. As a result, the browser couldn't validate the full digital certificate chain to ensure you were really connecting to the website you intended to connect to.

This is a common scenario on security incidents, where Man-in-the-Middle (MitM) attacks or direct web server breaches modify the SSL/TLS certificate offered to the victim, and when accidentally accepted, the attacker can intercept and modify the "secure" HTTPS channel. As you may find yourself dealing with a similar situation in the future... how can you (as I did) check what is the real reason behind the SSL/TLS certificate validation error? By manually verifying the SSL/TLS certificate trust chain, or certificate hierarchy, through openssl.

The goal is to manually follow all the validation steps that are commonly performed it an automatic way by the web browser.

Step 1: Check the certificate validation error and download the controversial digital certificate.

$ openssl s_client -connect isc.sans.org:443 depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org verify error:num=27:certificate not trusted verify return:1 depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org verify error:num=21:unable to verify the first certificate verify return:1 CONNECTED(00000003) --- Certificate chain 0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA --- Server certificate -----BEGIN CERTIFICATE----- MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/ MQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBD aXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxKjAoBgNVBAMTIVVT RVJUcnVzdCBMZWdhY3kgU2VjdXJlIFNlcnZlciBDQTAeFw0xMDAzMzEwMDAwMDBa Fw0xMTAzMzEyMzU5NTlaMIH5MQswCQYDVQQGEwJVUzEOMAwGA1UEERMFMjA4MTQx ETAPBgNVBAgTCE1hcnlsYW5kMREwDwYDVQQHEwhCZXRoZXNkYTESMBAGA1UECRMJ U3VpdGUgMjA1MRowGAYDVQQJExE4MTIwIFdvb2Rtb250IEF2ZTEbMBkGA1UEChMS VGhlIFNBTlMgSW5zdGl0dXRlMSgwJgYDVQQLEx9OZXR3b3JrIE9wZXJhdGlvbnMg Q2VudGVyIChOT0MpMSYwJAYDVQQLEx1Db21vZG8gVW5pZmllZCBDb21tdW5pY2F0 aW9uczEVMBMGA1UEAxMMaXNjLnNhbnMub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOC AQ8AMIIBCgKCAQEAvYLqphsusuyiVDookRrfsZDx0T1SOf1EQmzUFSK0qVTv+RKM VanUSwsbWuMYrS6W7bGld2/qCGuk2rxMjwtiNE1x2rtaXUkngoKfnQKH+zGErbmc Ead2IjY5OANT6xnxfJS8a29XTqRrKJW8c/PR9NYCGUmq1X+0EnB4+wPZbDsebZ3h 2wISxDaEJlOYbc3MzNyKwbTkwsyn1uwqZS0DQyKqVGL4hZeVmy5OwW9b8/RLYOov NYxTH0bHlbiaOuwakm4cx/ZJMELSBhbgwjt2sLkD8CShdES6fHPPAeMLW//ir3em RMXABEnF1wE5CmvMDe9e6b/joHurw34BISM8twIDAQABo4IB/DCCAfgwHwYDVR0j BBgwFoAUr6RAr58W/qsx/fvVl4v1kaMkhhYwHQYDVR0OBBYEFN6hBRDWEgv4hVZI hR1/CjtAx8k2MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQW MBQGCCsGAQUFBwMBBggrBgEFBQcDAjBCBgNVHSAEOzA5MDcGDCsGAQQBsjEBAgED BDAnMCUGCCsGAQUFBwIBFhlodHRwczovL2Nwcy51c2VydHJ1c3QuY29tMEsGA1Ud HwREMEIwQKA+oDyGOmh0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RM ZWdhY3lTZWN1cmVTZXJ2ZXJDQS5jcmwwfQYIKwYBBQUHAQEEcTBvMEYGCCsGAQUF BzAChjpodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0TGVnYWN5U2Vj dXJlU2VydmVyQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1 c3QuY29tMGkGA1UdEQRiMGCCDGlzYy5zYW5zLm9yZ4IRaXNjLmluY2lkZW50cy5v cmeCDGlzYy5zYW5zLmVkdYINaXNjMS5zYW5zLm9yZ4INaXNjMi5zYW5zLm9yZ4IR d3d3LmluY2lkZW50cy5vcmcwDQYJKoZIhvcNAQEFBQADggEBAGBhcG/MGgAsiwsB AAg4ZYndAEukeYXpIbXzU5T9ZqkqHPWiUearWDzBgWKJcpgF+v9ZCqCrKuYbXXnv zqZi+BPKX3BSMxHlDUZ0C6tU/G6+FqcV4j19S+xPjAVvk28yqaZc9BNpLnCogAc/ F3gHL7SwCDFowP+RaqUlbUr1UtQgKDILWSOM4ukoVMbCt5nNmyXu0eDFb9tlWkJK KdPYzuKCYfWS7/9bQ868fML3m1xCZqG7L0t/XAFSZYN3ytqtbRTyOjcjdEwSxwA5 O3gV+dW6qM7/AmGZu+0/Grw3WMwiXzO8tRAHYliNz9PDqnK2k5NE0VbKKhndsoRc oAA+AfY= -----END CERTIFICATE----- subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA --- No client certificate CA names sent --- SSL handshake has read 2233 bytes and written 325 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 2048 bit Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1 Cipher : DHE-RSA-AES256-SHA Session-ID: 08BED94D2BBA7E525FB37BFE20DCD155CE62C93871B41ABBDF810D663FFC4A61 Session-ID-ctx: Master-Key: 620F10AF948333D43BCC2656E4493563C4A827A8BFAD46AF0815CF3643C602C0E1EBA3CD5DBFE0C4BA65F2DBD9762DF2 Key-Arg : None Start Time: 1271777232 Timeout : 300 (sec) Verify return code: 21 (unable to verify the first certificate) --- closed

From the output, ans specifically the verify return code at the end, you can see that the server certificate cannot be verified.

First of all, create a "certs" directory to put all the required files in. Copy and paste to a file ("ISC.pem") the digital certificate, that is, the text between "-----BEGIN CERTIFICATE-----" to "-----END CERTIFICATE-----" (including both lines).

Step 2: Identify the issuer and get its certificate.

Open the "ISC.pem" certificate file (by double-clicking on it on most operating systems) and inspect the following fields:

• The certificate thumbprint or fingerprint that identifies the server certificate: "bd:95:df:ac...46:aa" (SHA1).
• Issuer (under the "Certificate" section): Who did generate and issue the server certificate? "USERTrust Legacy Secure Server CA" from "The USERTRUST Network".
• The "Certificate Authority Key Identifier" or fingerprint (under "Certificate - Extensions"): "af:a4:40:af...86:16".
• The "Authority Information Access" (under the same section): It contains a pointer to the digital certificate of the issuer certification authority (CA): "URI: http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt".

Obtain a copy of the issuer certificate. The most secure option would be to get its certificate through HTTPS and not HTTP, but this only depends on how the CA decided to make it available. Double check with the CA website that the URL and the fingerprint are valid. In this case, USERTrust was acquired by Comodo, and the issuer certificate is available here (https link) and referenced in its list of certificates. This certificate belongs to the USERTrust intermediate CA and was the one not available in Firefox 3.6.3 by default, hence, the root cause of the initial SSL/TLS error on the ISC website.

Although you might be tempted to perform the manual verification all from the command line, it is not the most secure option, as you could be forced to use http vs. https when using wget or curl. Depending on the version and platform of these tools, they may be distributed without a default list of trusted root certificates or do not use the list available on the system. Therefore, ** this is NOT the way to get the intermediate certificate **, use a web browser instead:

$ wget http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt --2010-04-20 17:32:44--  http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt ... 2010-04-20 17:32:45 (32.0 MB/s) - `USERTrustLegacySecureServerCA.crt' saved [1073/1073] $

Step 3: Try to verify the digital certificate again, but this time make use of the previously downloaded certificate ("USERTrustLegacySecureServerCA.crt").

Before using the downloaded certificate, we need to convert it to the PEM format (not required this time; exemplified later), and build the certificates directory required by the openssl "-CApath" option. The Unix "c_rehash" script helps to create the appropriate directory structure and certificate hash symbolic links. Be sure to rename all the certificates in PEM format to .pem, such as "USERTrustLegacySecureServerCA.crt":

$ c_rehash ./certs Doing ./certs ISC.pem => fc1aa8ab.0 USERTrustLegacySecureServerCA.pem => cf831791.0  $

If we try to validate the certificate again, and if we already have the certificates for all the intermediate and root CA's identified in the trust certificate chain stored on the "certs" directory, we will get a positive response: "Verify return code: 0 (ok)".

$ openssl s_client -CApath ./certs -connect isc.sans.org:443 CONNECTED(00000003) depth=2 /C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority verify return:1 depth=1 /C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA verify return:1 depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org verify return:1 --- Certificate chain  0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org    i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA --- Server certificate -----BEGIN CERTIFICATE----- MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/ ... oAA+AfY= -----END CERTIFICATE----- subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA --- No client certificate CA names sent --- SSL handshake has read 2233 bytes and written 325 bytes --- New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA Server public key is 2048 bit Compression: NONE Expansion: NONE SSL-Session:     Protocol  : TLSv1     Cipher    : DHE-RSA-AES256-SHA     Session-ID: C898C8DB5CD9CDFEE404451BA3E19A440951A1960DAC1BA62FD35F23D9772B30     Session-ID-ctx:     Master-Key: EC4D939A112112AAAB01DFF5FA0A5F6C26C568C8DEBBDF3A61515E8CD83F257DAB5894BC450A97A7EE5ABAB0B1893795     Key-Arg   : None     Start Time: 1271778616     Timeout   : 300 (sec)     Verify return code: 0 (ok) --- closed

If the certificate chain or hierarchy contains additional certificates, that is, there are multiple intermediate CA's involved, you may need to repeat the same process and download the certificates for all the other intermediate CA's and the root CA (omitted for brevity). For example, the intermediate USERTrust certificate was issued by "Entrust.net Secure Server Certification Authority". This root CA certificate can be manually obtained in DER format from Entrust website, with a fingerprint of "f0:17:62:13...d0:1a".

Once again, this DER file must be converted to PEM format using openssl:

$ openssl x509 -in entrust_ssl_ca.der -inform DER -outform PEM -out entrust_ssl_ca.pem

Finally, you will need to rebuild the certificates directory again, using "c_rehash", once it contains all the intermediate and root CA certificate files that belong to the certificate chain being tested, and try to verify the certificate again.

Part 2 of this article covers the chain layout for the ISC certificate in this case, how to identify the missing certificate on the web browser trust certificates list, and how to avoid this kind of error from the web server perspective, using Apache.