Akamai CloudFront Certificate pinning

We recently took over a project that has some static angularJS code hosted in via standard Route 53 -> CloudFront -> S3. And domain CDN is using Akamai which has origin set to the R53 url. Akamai has a `Specific Certificates (pinning)` setting to pull certificate from a *.cloudfront.net. 

Over the weekend something odd happened, some of our users can access, some get Akamai error, some get 404. Turns out the reason is the CloudFront certificate expires and amazon signed a new one. However in our Akamai side, we still hold that old cert’s hash which is used for ssl validation. As a result the ssl connection cannot be established. To solve the issue we have to re-extract the hash for new certificate in Akamai console. And it worked in our QA. The time we are ready to apply this change  to prod, our prod suddenly worked… We believe Akamai also have some mechanism to periodically pull the site for new certificate chain, or the max-age of the old hash reached.

A final solution will be create a new cert with R53 domain name to avoid all these pin/expire issue.

Certificate Pinning is basically allow us to ignore the regular SSL certificate chain verification process and just/only trust the provided certs. For Akamai, it generates the SHA-1 fingerprint of the leaf cert in the chain and use that to compare on each SSL handshake.

This link show chrome/Symantec is working on phasing out Symantec issued certs( Certs signed with Dec 1st 2017 or later are still good).


EMR hive JDBC over SSL with ELB

Recently we need to setup a hive cluster consuming S3 objects so that we could run query from our java server(tomcat) via JDBC.

Several challenges:

  1. our java server is on prem(will move to aws in 2017) so we have to secure the channel to the ERM cluster in emr.
    Solution: use SSL cross the board.
  2. Every time the EMR cluster restarts, its ip changes. For the JDBC connection, we would like a constant value (DNS) name.
    Solution: use ELB to obtain a DNS name.
  3. EMR master needs to be attached to ELB every time it is created
    Solution: in the bootstrap script, we constantly pull the state of the EMR creation state and once it finishes, grab its IP and attach to ELB.

For the challenge 1

we need to install the certificate chain as well as the private key into ELB so that it could accept ssl connection from client side. In the client JDBC, we need to add ‘;ssl=true’ to the jdbc connection string to instruct client to init the connection with SSL. Another thing is to import the CA to the JRE’s lib/security cacerts so that when we do SSL handshake, the ELB’s certificate CA would be in the java client’s truststore.  For our test, we use a self-CA(finrarcselfca.crt is our self CA certificate):

sudo keytool -keystore cacerts -importcert -alias finrarcselfca -file finrarcselfca.crt

As for the connection between ELB and EMR master, we can just use a random self-signed keystore since ELB does not need to verify the ERM cluster.First generate a self-signed keystore using:

keytool -genkey -keyalg RSA -alias selfsigned -keystore self.keystore -storepass changeit -validity 360 -keysize 2048

and then add the below config to the hive-site.xml


Note: the password should match for the one when created and the one specifed in the hive site xml.

For the challenge 2

In the ELB listener, we can forward the SSL TCP 443 port to the SSL TCP 10000 port. This way, when our java client init the JDBC over SSL connection, the ELB could unwrap the message and then start another SSL connection with the EMR master via port 10000.

For the challenge 3

Previously We have following shell script to wait ERM startup and attach to ELB. The bash way is quick and dirty.
Now we are leverage nodejs with aws-js-sdk to maintain our cluster which is more robust and easier to maintain/understand.

if [ $exitStatus -eq 0 ]
   clusterIdLine=`echo $result| sed 's/{\|"\|:\|}\|ClusterId//g'`
   clusterId=`echo $clusterIdLine| awk '{$1=$1;print}'`
   Mail "Adhoc Query Cluster $clusterId started..."
   Mail "Error while creating Adhoc Query Cluster $result"

sleep 30

while :
clusterState=`aws emr list-clusters --active |grep '"Id":\|"State":' | sed 'N;s/\n/ /' |grep "$clusterId" |sed 's/"\|,\|Name\|Id//g'|cut -d':' -f2|awk '{$1=$1;print}'`
echo $clusterId
echo $clusterState
if [ $clusterState = 'WAITING' ]
  echo 'Waiting for cluster to be created'
  sleep 30

masterInstanceID=`aws emr list-instances --cluster-id $clusterId --instance-group-types MASTER |grep '"Ec2InstanceId":' | sed 'N;s/\n/ /' |sed 's/"\|,\|Ec2InstanceId//g'|cut -d':' -f2|awk '{$1=$1;print}'`

echo $masterInstanceID

result=`aws elb register-instances-with-load-balancer --load-balancer-name $elbName --instances $masterInstanceID`

echo $result

Some reference

How to Create a Self Signed Certificate using Java Keytool

import private key to keystore

Cloudera hive config

How SSL works

Simple version

  1. Your web browser downloads the web server’s certificate, which contains the public key of the web server. This certificate is signed with the private key of a trusted certificate authority.
  2. Your web browser comes installed with the public keys of all of the major certificate authorities. It uses this public key to verify that the web server’s certificate was indeed signed by the trusted certificate authority.
  3. The certificate contains the domain name and/or ip address of the web server. Your web browser confirms that the address listed in the certificate is the one to which it has an open connection.
  4. Your web browser generates a shared symmetric key which will be used to encrypt the HTTP traffic on this connection; this is much more efficient than using public/private key encryption for everything. Your browser encrypts the symmetric key with the public key of the web server then sends it back, thus ensuring that only the web server can decrypt it, since only the web server has its private key.

Note that the certificate authority (CA) is essential to preventing man-in-the-middle attacks. However, even an unsigned certificate will prevent someone from passively listening in on your encrypted traffic, since they have no way to gain access to your shared symmetric key.


SSL协议使用不对称加密技术实现会话双方之间信息的安全传递。可以实现信息传递的保密性、完整性,并且会话双方能鉴别对方身份。不同于常用的http协议,我们在与网站建立SSL安全连接时使用https协议,即采用https://ip:port/的方式来访问。当我们与一个网站建立https连接时,我们的浏览器与Web Server之间要经过一个握手的过程来完成身份鉴定与密钥交换,从而建立安全连接。具体过程如下:
客户端检查服务器证书,如果检查失败,提示不能建立SSL连接。如果成功,那么继续。客户端浏览器为本次会话生成pre-master secret,并将其用服务器公钥加密后发送给服务器。如果服务器要求鉴别客户身份,客户端还要再对另外一些数据签名后并将其与客户端证书一起发送给服务器。
如果服务器要求鉴别客户身份,则检查签署客户证书的CA是否可信。如果不在信任列表中,结束本次会话。如果检查通过,服务器用自己的私钥解密收到的pre-master secret,并用它通过某些算法生成本次会话的master secret。
客户端与服务器均使用此master secret生成本次会话的会话密钥(对称密钥)。在双方SSL握手结束后传递任何消息均使用此会话密钥。这样做的主要原因是对称加密比非对称加密的运算量低一个数量级以上,能够显著提高双方会话时的运算速度。


 Complete Version

To see a complete worked example of this process using Firefox connecting to amazon.com, seemoserware.com/2009/06/first-few-milliseconds-of-https.html


cert = identity_info + public key + digit signature(CA signed identity-info)

my another post in Chinese about cert.

4 questions need to be address during the authentication:

  1. Has the Digital Certificate been issued/signed by a Trusted CA?
    in browser, already CA installed. In java, import others from `cacert` file
  2. Is the Certificate Expired – checks both the start and end dates
    make sure it is not expired
  3. Has the Certificate been revoked? (Could be OCSP or CRL check)
    either from a Certificate Revocation List (CRL), OR from a Online Certificate Status Protocol (OCSP).
  4. Has the client provided proof of possession?
    sign the message’s hash with its private key and send along with the cert so that the receiver could use the public key to decrypt the message hash and do a hash to the message and then compare.

Client certificate(2 way SSL)

Client also need cert from CA(same or another) as well as the private key. Server need the CA which signed the client cert in its cacert.

During the handshake, the client not only send the cert to the server, but also something signed by its private key. More specific: according to RFC5246 7.4.8.  Certificate Verify  in which client sign the handshake messages in the `Certificate Verify` message of the TLS handshake so that the server can verify it against the public key sent in the client certificate. Without this step, no client-certificate authentication would be taking place.





  数字证书是一个经证书认证中心(CA)数字签名的包含公开密钥拥有者信息以及公开密钥的文件。认证中心(CA)作为权威的、可信赖的、公正的第三方机构,专门负责为各种认证需求提供数字证书服务。认证中心颁发的数字证书均遵循X.509 V3标准。X.509标准在编排公共密钥密码格式方面已被广为接受。X.509证书已应用于许多网络安全,其中包括IPSec(IP安全)、SSL、SET、S/MIME。














(8)乙用甲的公钥(PK)对甲的数字签名进行解密,得到信息摘要。 乙用相同的hash算法对收到的明文再进行一次hash运算,得到一个新的信息摘要。



HERE is a good series of how to secure webservice