Blog

21
January 2014

Gavin Pickin

Techie Gotcha - SSL Certificate Problems with Apache and Issuer Chain

Apache, OpenSSL, Server Admin, Techie Gotchas

I am sure SSL Certificates are not new to most Web Developers, unless you have the luxury of an Admin team. I have been using them for years, most of those years admittedly on Windows, but the last several years we have been migrating all of our windows boxes over to Linux (as you could tell from the majority of my posts). Just recently though, something new came up, which I had not seen before, so I thought I would share my experience, for anyone else looking for a solution out there.

A customer had called, and stated that their Security Certificate was not working correctly on their site. We had just renewed it, installed it, and tested it. So I do the usual tests, I pull it up in my browser, check for http calls on an https page, check the certificate information, all looks good. Then we check with the customer, and they tell us they are using Firefox. My default browser is chrome, so I reach for Firefox and sure enough there are SSL problems with the cert. Firefox shows the following error message.

www.domainname.com uses an invalid security certificate.
The certificate is not trusted because no issuer chain was provided.
(Error code: sec_error_unknown_issuer)

Interesting, isn't it? 

I checked Safari Chrome and Opera, all good, and apparently IE is the same, but Firefox, as of version 23/24 seems to have added an additional security check, where Firefox checks the whole issuer chain. Apparently IE and some browsers automatically download the Chain for you behind the scenes, if possible, but Firefox cannot or does not.

Now, one thing I noticed was none of my SSL certificates on my Windows boxes have any ssl issues, so I assume that IIS on those machines takes care of the issuer chain. I know when you download an SSL Certificate you get the Certificate File and a Bundle… so obviously that is used for exactly this. 

To verify what is actually going on, I used a cool little tool from SSLShopper.com… which is their SSL Checker.
http://www.sslshopper.com/ssl-checker.html

When you run it for one of the domains we host (tweaked to reproduce the issue for this article) you will see something like this.

If you scroll down, you will see the SSL Cert and the Chain of Issuers, and in this case, its broken, as you can see below.

The main reason I'm blogging this, is because all of the information i found all pointed to Mozilla, and has a lot of information on how someone should fix their browser. Looking at this information, its not a browser issue really, there really is an issue with the way the SSL is installed and being used. 

I did some more digging, and found more information about how to set your SSL in your Apache Configuration files with OpenSSL… and noticed, I was missing one particular field.

SSLEngine On
SSLCertificateFile /PathToMySSLs/2014.www.donlucas.com.cert
SSLCertificateKeyFile /PathToMySSLs/2014.www.donlucas.com.key
SSLCertificateChainFile PathToMySSLs/intermediates/sf_bundle-g2-g1.crt

The last line above was missing in this particular virtual host, so I downloaded the bundle file with the Cert again. Stored the Bundle file in an Intermediates folder and added the line to my Virtual Host, and reloaded my Apache Httpd Config and reloaded the domain in Firefox, and success.

I check the site again in SSL Shopper and I see the following result.
Now the Certificate has been listed correctly

If we scroll down, we see the full chain, and this time, there is no broken links.

So that seemed simple enough, but something was still bothering me. I know I had seen the SSLCertificateChainFile configuration before, so I decided to go through my other SSL certs setup in Apache with OpenSSL… and I realized, some of our sites were using that configuration, and setup to use sf_bundle.crt. Why were those domain names failing though? 

It looks like the more recent renewals were using a different Intermediate Bundle to complete the Issuer chain.

 

What does this mean? 

This means even if you have been and were using the SSLCertificateChainFile config on your virtual hosts, you need to ensure you check the Issuer Chain with a tool like SSL Shoppers' SSL Check, to ensure the Intermediate Bundle you were using, is still valid

Of course, you can download the Bundle and install it each and every time, but I'm not sure if the SSL Providers are going to name them enough to be able to differentiate versions.

 

So why is Firefox different?

Some say because of SSL Certificate vulnerabilities of providers like Comodo, Firefox wants to err on the side of being more secure, and enforce the Server to install and reference all the Intermediate Bundles, because its seen as a security problem.

There is great debate on Mozilla forums (just one of the many links here) for and against this stand… interesting that they are the only browser not downloading the Intermediate Certs automatically.

SSL Shopper has more information on this topic here
http://www.sslshopper.com/ssl-certificate-not-trusted-error.html

I do not know how helpful this is, but blogging it helps me log my work, and I'm sure someone will come across this type of issue at some point.

Thanks for reading,

Gavin

by Mark
03/04/2014 10:29:03 AM

Gavin-

THANK YOU! THANK YOU! THANK YOU! We ran into the issue with a new Go Daddy cert as well. It was discovered on curl commands hitting our api. We could then only find the issue on Firefox. But there was still a real issue! Your article articulated everything I was experiencing and helped us to sove. However, the new bundle from Go Daddy is not as updated as all the individual components and we had to append them all to get it to work. Don't always rely on the bundle when still experiencing issues. I appreciated the time put forth to share your discovery. You helped me immensely. Much gratitude.

-Mark

by danil
05/16/2014 11:22:03 AM

Great post sslchecker was much apprciated

by grateful
10/17/2014 01:07:17 PM

Thank you for unraveling this gotcha! Just hit it trying to install a new cert. Bet you get a lot more hits as sites transition from SHA1 to SHA2 due to SHA1's deprecation.

by Kata
01/16/2015 07:22:15 AM

Very helpful and informative, thanks!

by Gilbert
04/28/2015 11:08:15 AM

Hi, Just to tell you your post helped me after many hours of search. Thanks

Blog Search