Saturday, April 24, 2010

Manual Verification of SSL/TLS Certificate Trust Chains using Openssl (Part 1/2)

This week, during my Internet Storm Center (ISC) shift, Firefox 3.6.3 (the latest available version) displayed a digital certificate error when accessing the ISC login page through SSL/TLS: https://isc.sans.org/myisc.html. I confirmed this on a couple of Firefox instances running on Mac OS X and Windows XP.


We also got a few reports from ISC readers on the same issue, although other people running the same browser version, and even language (EN), on the same OS platforms, didn't get any error message. Finally, the reason was a new ISC digital certificate had been recently installed, and the required intermediate certificate was missing in some web browsers. As a result, the browser couldn't validate the full digital certificate chain to ensure you were really connecting to the website you intended to connect to.

This is a common scenario on security incidents, where Man-in-the-Middle (MitM) attacks or direct web server breaches modify the SSL/TLS certificate offered to the victim, and when accidentally accepted, the attacker can intercept and modify the "secure" HTTPS channel. As you may find yourself dealing with a similar situation in the future... how can you (as I did) check what is the real reason behind the SSL/TLS certificate validation error? By manually verifying the SSL/TLS certificate trust chain, or certificate hierarchy, through openssl.

The goal is to manually follow all the validation steps that are commonly performed it an automatic way by the web browser.

Step 1: Check the certificate validation error and download the controversial digital certificate.

$ openssl s_client -connect isc.sans.org:443
depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
verify error:num=21:unable to verify the first certificate
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/
MQswCQYDVQQGEwJVUzELMAkGA1UECBMCVVQxFzAVBgNVBAcTDlNhbHQgTGFrZSBD
aXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxKjAoBgNVBAMTIVVT
RVJUcnVzdCBMZWdhY3kgU2VjdXJlIFNlcnZlciBDQTAeFw0xMDAzMzEwMDAwMDBa
Fw0xMTAzMzEyMzU5NTlaMIH5MQswCQYDVQQGEwJVUzEOMAwGA1UEERMFMjA4MTQx
ETAPBgNVBAgTCE1hcnlsYW5kMREwDwYDVQQHEwhCZXRoZXNkYTESMBAGA1UECRMJ
U3VpdGUgMjA1MRowGAYDVQQJExE4MTIwIFdvb2Rtb250IEF2ZTEbMBkGA1UEChMS
VGhlIFNBTlMgSW5zdGl0dXRlMSgwJgYDVQQLEx9OZXR3b3JrIE9wZXJhdGlvbnMg
Q2VudGVyIChOT0MpMSYwJAYDVQQLEx1Db21vZG8gVW5pZmllZCBDb21tdW5pY2F0
aW9uczEVMBMGA1UEAxMMaXNjLnNhbnMub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAvYLqphsusuyiVDookRrfsZDx0T1SOf1EQmzUFSK0qVTv+RKM
VanUSwsbWuMYrS6W7bGld2/qCGuk2rxMjwtiNE1x2rtaXUkngoKfnQKH+zGErbmc
Ead2IjY5OANT6xnxfJS8a29XTqRrKJW8c/PR9NYCGUmq1X+0EnB4+wPZbDsebZ3h
2wISxDaEJlOYbc3MzNyKwbTkwsyn1uwqZS0DQyKqVGL4hZeVmy5OwW9b8/RLYOov
NYxTH0bHlbiaOuwakm4cx/ZJMELSBhbgwjt2sLkD8CShdES6fHPPAeMLW//ir3em
RMXABEnF1wE5CmvMDe9e6b/joHurw34BISM8twIDAQABo4IB/DCCAfgwHwYDVR0j
BBgwFoAUr6RAr58W/qsx/fvVl4v1kaMkhhYwHQYDVR0OBBYEFN6hBRDWEgv4hVZI
hR1/CjtAx8k2MA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjBCBgNVHSAEOzA5MDcGDCsGAQQBsjEBAgED
BDAnMCUGCCsGAQUFBwIBFhlodHRwczovL2Nwcy51c2VydHJ1c3QuY29tMEsGA1Ud
HwREMEIwQKA+oDyGOmh0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9VU0VSVHJ1c3RM
ZWdhY3lTZWN1cmVTZXJ2ZXJDQS5jcmwwfQYIKwYBBQUHAQEEcTBvMEYGCCsGAQUF
BzAChjpodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0TGVnYWN5U2Vj
dXJlU2VydmVyQ0EuY3J0MCUGCCsGAQUFBzABhhlodHRwOi8vb2NzcC51c2VydHJ1
c3QuY29tMGkGA1UdEQRiMGCCDGlzYy5zYW5zLm9yZ4IRaXNjLmluY2lkZW50cy5v
cmeCDGlzYy5zYW5zLmVkdYINaXNjMS5zYW5zLm9yZ4INaXNjMi5zYW5zLm9yZ4IR
d3d3LmluY2lkZW50cy5vcmcwDQYJKoZIhvcNAQEFBQADggEBAGBhcG/MGgAsiwsB
AAg4ZYndAEukeYXpIbXzU5T9ZqkqHPWiUearWDzBgWKJcpgF+v9ZCqCrKuYbXXnv
zqZi+BPKX3BSMxHlDUZ0C6tU/G6+FqcV4j19S+xPjAVvk28yqaZc9BNpLnCogAc/
F3gHL7SwCDFowP+RaqUlbUr1UtQgKDILWSOM4ukoVMbCt5nNmyXu0eDFb9tlWkJK
KdPYzuKCYfWS7/9bQ868fML3m1xCZqG7L0t/XAFSZYN3ytqtbRTyOjcjdEwSxwA5
O3gV+dW6qM7/AmGZu+0/Grw3WMwiXzO8tRAHYliNz9PDqnK2k5NE0VbKKhndsoRc
oAA+AfY=
-----END CERTIFICATE-----
subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 2233 bytes and written 325 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES256-SHA
Session-ID: 08BED94D2BBA7E525FB37BFE20DCD155CE62C93871B41ABBDF810D663FFC4A61
Session-ID-ctx:
Master-Key: 620F10AF948333D43BCC2656E4493563C4A827A8BFAD46AF0815CF3643C602C0E1EBA3CD5DBFE0C4BA65F2DBD9762DF2
Key-Arg : None
Start Time: 1271777232
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
closed

From the output, ans specifically the verify return code at the end, you can see that the server certificate cannot be verified.

First of all, create a "certs" directory to put all the required files in. Copy and paste to a file ("ISC.pem") the digital certificate, that is, the text between "-----BEGIN CERTIFICATE-----" to "-----END CERTIFICATE-----" (including both lines).

Step 2: Identify the issuer and get its certificate.

Open the "ISC.pem" certificate file (by double-clicking on it on most operating systems) and inspect the following fields:
  • The certificate thumbprint or fingerprint that identifies the server certificate: "bd:95:df:ac...46:aa" (SHA1).
  • Issuer (under the "Certificate" section): Who did generate and issue the server certificate? "USERTrust Legacy Secure Server CA" from "The USERTRUST Network".
  • The "Certificate Authority Key Identifier" or fingerprint (under "Certificate - Extensions"): "af:a4:40:af...86:16".
  • The "Authority Information Access" (under the same section): It contains a pointer to the digital certificate of the issuer certification authority (CA): "URI: http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt".
Obtain a copy of the issuer certificate. The most secure option would be to get its certificate through HTTPS and not HTTP, but this only depends on how the CA decided to make it available. Double check with the CA website that the URL and the fingerprint are valid. In this case, USERTrust was acquired by Comodo, and the issuer certificate is available here (https link) and referenced in its list of certificates. This certificate belongs to the USERTrust intermediate CA and was the one not available in Firefox 3.6.3 by default, hence, the root cause of the initial SSL/TLS error on the ISC website.

Although you might be tempted to perform the manual verification all from the command line, it is not the most secure option, as you could be forced to use http vs. https when using wget or curl. Depending on the version and platform of these tools, they may be distributed without a default list of trusted root certificates or do not use the list available on the system. Therefore, ** this is NOT the way to get the intermediate certificate **, use a web browser instead:

$ wget http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt
--2010-04-20 17:32:44--  http://crt.usertrust.com/USERTrustLegacySecureServerCA.crt
...
2010-04-20 17:32:45 (32.0 MB/s) - `USERTrustLegacySecureServerCA.crt' saved [1073/1073]
$

Step 3: Try to verify the digital certificate again, but this time make use of the previously downloaded certificate ("USERTrustLegacySecureServerCA.crt").

Before using the downloaded certificate, we need to convert it to the PEM format (not required this time; exemplified later), and build the certificates directory required by the openssl "-CApath" option. The Unix "c_rehash" script helps to create the appropriate directory structure and certificate hash symbolic links. Be sure to rename all the certificates in PEM format to .pem, such as "USERTrustLegacySecureServerCA.crt":

$ c_rehash ./certs
Doing ./certs
ISC.pem => fc1aa8ab.0
USERTrustLegacySecureServerCA.pem => cf831791.0 
$

If we try to validate the certificate again, and if we already have the certificates for all the intermediate and root CA's identified in the trust certificate chain stored on the "certs" directory, we will get a positive response: "Verify return code: 0 (ok)".

$ openssl s_client -CApath ./certs -connect isc.sans.org:443
CONNECTED(00000003)
depth=2 /C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority
verify return:1
depth=1 /C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA
verify return:1
depth=0 /C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
verify return:1
---
Certificate chain
 0 s:/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGATCCBOmgAwIBAgIQOxCOI6FirgnSgN/fCRi57jANBgkqhkiG9w0BAQUFADB/
...
oAA+AfY=
-----END CERTIFICATE-----
subject=/C=US/postalCode=20814/ST=Maryland/L=Bethesda/streetAddress=Suite 205/streetAddress=8120 Woodmont Ave/O=The SANS Institute/OU=Network Operations Center (NOC)/OU=Comodo Unified Communications/CN=isc.sans.org
issuer=/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/CN=USERTrust Legacy Secure Server CA
---
No client certificate CA names sent
---
SSL handshake has read 2233 bytes and written 325 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 2048 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: C898C8DB5CD9CDFEE404451BA3E19A440951A1960DAC1BA62FD35F23D9772B30
    Session-ID-ctx:
    Master-Key: EC4D939A112112AAAB01DFF5FA0A5F6C26C568C8DEBBDF3A61515E8CD83F257DAB5894BC450A97A7EE5ABAB0B1893795
    Key-Arg   : None
    Start Time: 1271778616
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

---
closed

If the certificate chain or hierarchy contains additional certificates, that is, there are multiple intermediate CA's involved, you may need to repeat the same process and download the certificates for all the other intermediate CA's and the root CA (omitted for brevity). For example, the intermediate USERTrust certificate was issued by "Entrust.net Secure Server Certification Authority". This root CA certificate can be manually obtained in DER format from Entrust website, with a fingerprint of "f0:17:62:13...d0:1a".

Once again, this DER file must be converted to PEM format using openssl:

$ openssl x509 -in entrust_ssl_ca.der -inform DER -outform PEM -out entrust_ssl_ca.pem

Finally, you will need to rebuild the certificates directory again, using "c_rehash", once it contains all the intermediate and root CA certificate files that belong to the certificate chain being tested, and try to verify the certificate again.

Part 2 of this article covers the chain layout for the ISC certificate in this case, how to identify the missing certificate on the web browser trust certificates list, and how to avoid this kind of error from the web server perspective, using Apache.

2 comments:

jors said...

Just a note on the 'magic' of double-clicking a certificate to inspect its fields: on GNU/Linux, certificate viewers/handlers could be kleopatra (for KDE) and gnomint (for Gnome). Cheers.

Glenn Goodrich said...

So, this post just helped me a TON with JBoss/Torquebox. THANKS!

Post a Comment