Akamai Origin ERR_FWD_SSL_HANDSHAKE

Background

We recently took a project from another team which has CICD pipeline setup that creates a CloudFront(CF) Distro each time and direct traffic for static content and api from there based on path. The static content is fronted via Akamai then route 53(r53) then CF. CF contains the r53 domains as cname, however without a real cert backing them.
Akamai -> root r53 -> dynamic r53 -> CF

New AWS CF policy

AWS CF had an announcement on Apr 2019 that going forward, all cnames added to CF must have corresponding certs covering the CN. This obviously breaks the above existing pipeline as the add r53 cname part will fail.

Prod down

So we applied a new cert for the root r53 we have which is also what Akamai is pointing to and deployed to CF by changing our CloudFormation templates. Then we created a new stack and switched the root r53 pointing to a newly created dynamic r53. And got some Akamai ERR_FWD_SSL_HANDSHAKE and site was down! What’s worse is due to the above CF policy change, we can not rollback as the old stack does not have valid certs for neither of the r53 records.

After 2 hours of trying different fixes like remove the middle r53 and have the root pointing to CF directly, pinned the new cert in Akamai to make sure akamai accepts it etc etc. Nothing worked. Eventually we have to point Akamai directly to the CF and have the CF using that wildcard Amazon managed cert to have the site up. No doubt, a dirty temporary solution.

Troubleshot

As bad as the temp solution is, it gave us a good opportunity to really test our root r53 and ssl combination freely as it is not used anywhere. First step is following the akamai staging test steps, modify the /etc/hosts file so that we can have our local traffic going to akamai staging servers. One caveat here is I have to do this in public internet rather than behind the corporate proxy which does not honor the hosts file at all.

What’s odd is the site(root r53) show fine in all browsers which means the browser venders all accept the certs and consider it as valid. Tried all the steps including tweaking CF configs mimic a current working stack, changing akamai configs on match CN/SAN etc. Nothing worked. Based on the error message, it must be related to SSL, so I tried to use openssl to verify the cert chain. openssl s_client -showcerts -verify 10 -connect locations.aws-rb.capitalonegslbex.com:443. and getting the error message: no peer certificate available.
Seems that server requires SNI so adding -servername locations.aws-rb.capitalonegslbex.com to the above command works.
This gives a good hint that we did not turn on the Use SNI TLS Extension in Akamai config. After switch that toggle, everything worked. So the conclusion here is CF does not return certs if no SNI header is provided. What a lesson…

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s