We recently took a project from another team which has CICD pipeline setup that creates a CloudFront(CF) Distro each time and direct traffic for static content and api from there based on path. The static content is fronted via Akamai then route 53(r53) then CF. CF contains the r53 domains as
cname, however without a real cert backing them.
Akamai -> root r53 -> dynamic r53 -> CF
New AWS CF policy
AWS CF had an announcement on Apr 2019 that going forward, all cnames added to CF must have corresponding certs covering the CN. This obviously breaks the above existing pipeline as the add r53 cname part will fail.
So we applied a new cert for the root r53 we have which is also what Akamai is pointing to and deployed to CF by changing our CloudFormation templates. Then we created a new stack and switched the root r53 pointing to a newly created dynamic r53. And got some Akamai
ERR_FWD_SSL_HANDSHAKE and site was down! What’s worse is due to the above CF policy change, we can not rollback as the old stack does not have valid certs for neither of the r53 records.
After 2 hours of trying different fixes like remove the middle r53 and have the root pointing to CF directly, pinned the new cert in Akamai to make sure akamai accepts it etc etc. Nothing worked. Eventually we have to point Akamai directly to the CF and have the CF using that wildcard Amazon managed cert to have the site up. No doubt, a dirty temporary solution.
As bad as the temp solution is, it gave us a good opportunity to really test our root r53 and ssl combination freely as it is not used anywhere. First step is following the akamai staging test steps, modify the
/etc/hosts file so that we can have our local traffic going to akamai staging servers. One caveat here is I have to do this in public internet rather than behind the corporate proxy which does not honor the
hosts file at all.
What’s odd is the site(root r53) show fine in all browsers which means the browser venders all accept the certs and consider it as valid. Tried all the steps including tweaking CF configs mimic a current working stack, changing akamai configs on
match CN/SAN etc. Nothing worked. Based on the error message, it must be related to SSL, so I tried to use
openssl to verify the cert chain.
openssl s_client -showcerts -verify 10 -connect locations.aws-rb.capitalonegslbex.com:443. and getting the error message:
no peer certificate available.
Seems that server requires
SNI so adding
-servername locations.aws-rb.capitalonegslbex.com to the above command works.
This gives a good hint that we did not turn on the
Use SNI TLS Extension in Akamai config. After switch that toggle, everything worked. So the conclusion here is CF does not return certs if no SNI header is provided. What a lesson…