lambda@edge prototype

Recently I was doing a MVP for replacing a ELB/EC2/Docker based static site preview stack with a cloudfront/lambda/s3 based one. Background The purpose of this is to reduce the maintenance we has to do with the EC2 stack like regular AMI update. reduce the complexity of the stack  as the previous one involves building custom…

nginx reverse proxy S3 files

China access issue Recently some of our church site users reported that the sermon audio/video download feature does not work any more. We recently moved our large files from file system to s3. After some research looks like the aws s3 is blocked by the famous Chinese Great FireWall(GFW). Possible Solutions Moving files back to…

aws cli ProfileNotFound

I was trying to do some KMS encryption for some of our prod credentials with aws cli. After pulling down the temporary aws sts token for prod roles and run the aws –profile SOME_PROD_ROLE kms encrypt xxx, the  botocore.exceptions.ProfileNotFound: The config profile (SOME_DEV_ROLE) could not be found constantly pop up. I checked the ~/.aws/credentails file and make…

Serverless EMR Cluster monitoring with Lambda

Background One issue we typically have is our EMR cluster stops consuming hive queries due to the overload of the metastore loading/refreshing. This is partially caused by the usage of the shared-metastore which hosts many teams’ schema/tables inside our organization. When this happens in prod, we have to ask help from RIM to terminate our…

stateful firewall with inbound outbound traffic

Background I have worked as Devops for cloud migration in the recent 3 months without really writing much code. Even though being exposed to many AWS services like EMR/EC2/ASG(auto scaling group)/LC(launch config)/CF(cloud formation) etc.. with the need of setting up security groups(SG), i find myself still a bit confusing with inbound and outbound traffic rules….

Add Username password auth to Hive

In my previous post, we achieved end to end SSL encryption from client to ELB to the EMR master. Our next requirement is to add username password authentication. There are different ways in hive to do this: 1. LDAP, 2. PAM, 3. CUSTOM mode. After some evaluation we finally choosed the  CUSTOM mode way. We…

EMR hive JDBC over SSL with ELB

Recently we need to setup a hive cluster consuming S3 objects so that we could run query from our java server(tomcat) via JDBC. Several challenges: our java server is on prem(will move to aws in 2017) so we have to secure the channel to the ERM cluster in emr. Solution: use SSL cross the board….

partition key, composite key and clustering key in Cassandra

There is a lot of confusion around this, I will try to make it as simple as possible. The primary key is a general concept to indicate one or more columns used to retrieve data from a Table. The primary key may be SIMPLE create table stackoverflow ( key text PRIMARY KEY, data text );…

deploy nodejs angularjs mongodb expressjs application to openshift

In my previous post, I described how to upload file using nodejs and angularjs. Now we are to deploy this MEAN stack app to openshift which is a very good cloud service provider offering 3 application deployment for free. You can even deploy Java web application to it using Tomcat/Mysql, part of which i mentioned…

add google code as a remote in git for intellij

add remote for intellij I use intellij to create a local git repo. To add a new remote: 1. create a project in google code. 2. go to this link: https://code.google.com/hosting/settings. Copy the machine code.google.com login xxxxx@gmail.com password xxxxxxxxx  to the user directory’s .netrc file: ~/.netrc then change the priv for this file, could be 500/600, i…