base64 in shell

Used to go to some site when in need of en/de-code from/to base64. Turns out we can do that from the shell directly.


echo -n 'My TextTo Encode' | base64

The -n is to prevent the new line of the echo from being included in the result(by default echo outputs a trailing newline).


echo 'My Encoded TExt | base64 --decode.

we can also use -D(MacOS) or -d(Linux) instead of --decode for short.


clipboard in terminal(copy/paste)

Was watching a CD video with Semaphore and saw the guy in the video manipulating clipboard with pipe directly in the terminal. Pretty cool! Did some research on that.


In Mac, it is pretty straightforward with the built in pbcopy and pbpaste. (pasteboard copy / paste). Some examples:

cat | pbcopy

pbpaste >

ps aux | grep myKeyWord | pbcopy

pbcopy < file.txt

pbpaste | grep myKeyWord


Take Ubuntu as example here. We can install a 16 kB program called xclip:

sudo apt-get install -y xclip

Then we can set alias so we can do the above operation like in osx. If you use bash then set your ~/.bashrc:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

If you are like me using fish as main shell, then similarly add below to your ~/.config/fish/ file.

alias pbcopy "xclip -selection clipboard"
alias pbpaste "xclip -selection clipboard -o"

bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.


A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

install ora2pg on mac osx

Recently we need to migrate our on prem oracle database into postgresql on aws RDS. Looks like ora2pg is a nice little tool to mitigate the problem though not perfect.

I am running OS X 10.10.5 Yosemite. It already comes with perl v5.18.2 as well as cpan for installing packages.

download/install Oracle client

The official document specifies quite vaguely about what is the oracle client libraries.  Turns out we need to download 3 packages from oracle website, for my use case we have oracle 12c so my current version is 12.1.02. The packages you need are:

  1. Oracle Basic
  2. Sqlplus
  3. sdk

Then we unzip them to the SAME directory like: /Users/liha/develop/instantclient_12_1 .

Now we need to set 2 env variables for installation:

export LD_LIBRARY_PATH=/Users/liha/develop/instantclient_12_1
export ORACLE_HOME=/Users/liha/develop/instantclient_12_1

install DBI/DBD

Now we need to install DBI/DBD stuff that is required during the ora2pg Makefile phase, especially the DBD::Oracle. The DBD:Pg is optional, only if you need to dump data/schema live rather than offline to file.

#perl –MCPAN –e shell
cpan> install DBI
cpan> install DBD::Oracle
cpan> install DBD::Pg

Check module

Now we can check whether the dependencies are correctly installed by running a perl script.

use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();

result should look like something:

➜  ora2pg perl                                                                                                                                                                              12:48:46

install ora2Pg

now can start to install the real thing by download and extract the ora2Pg tar.bz2 and run its which will eventually generate our makefile which we can make and install:

perl Makefile.PL
make && make install

After installation, the default conf file will be under etc/ora2pg/ora2pg.conf.

ora2Pg config

Now we can start config the ora2Pg by changing the default conf file or create a new conf file based on default conf. If you have multiple DB schemas to migrate or have multiple environments, it would be better to create different conf for each use case.

The important fields are:

ORACLE_DSN      dbi:Oracle:host=hostname;sid=Oracle SID

ORACLE_USER     schema_owner name

ORACLE_PWD      password

You should set as below to connect and retrieve required schema data and objects


I also turned debug on to view progress

DEBUG           1

Other settings used were


SCHEMA          schema_owner_name


I also used the same schema name in PG

PG_SCHEMA       schema_owner_name

I did have some problems with characters not being copied across as required and had to use settings below to override defaults, on some occasions



Now you can run the dump:

ora2pg -c myOra2pg.conf -t TABLE -o tables.sql


Shebang/Hashbang #! in shell

In unix sh file, we usually see the first line start with #!.

Like #!/bin/bash

The shebang (#!) at the start means execute the script with what follows.

In nodejs script, we usually see

#!/bin/env node

So this means the env command will try to find where the node has been installed from the $PATH, so that we do not have to specify the exact node path which would affect portability.


Spring boot jar process management in AWS EC2 instance with supervisor


Spring boot application is built into a jar which contains its own tomcat. So instead of running it in a traditional way of  having a tomcat instance that serving one or multiple wars, we will run the jar with java -jar. The problem here is if we run it directly, when our session quits/expires, the process will end. So we can either run it as a service(init.d), or using some 3rd party tool to manage it. In NodeJs world, we have the super powerful pm2. in the native world, ‘supervisor‘ seems to be the most recommended tool. Here we are going to introduce how to manage our spring boot jar with supervisor.

Install supervisor

Some EC2 AMI comes with easy_install’, which is a feature of setuptools. This is the preferred method of installation.

easy_install supervisor


Depending on the permissions of your system’s Python, you might need to be the root user to install Supervisor successfully using easy_install

Or we can use pip to install it thru: 

pip install supervisor

More reference in their official doc.

Supervisor Config

The Supervisor configuration file is conventionally named supervisord.conf. It is used by both supervisord and supervisorctl. If either application is started without the -coption (the option which is used to tell the application the configuration filename explicitly), the application will look for a file named supervisord.conf within the following locations, in the specified order. It will use the first file it finds.

  1.  $CWD/supervisord.conf
  2.  $CWD/etc/supervisord.conf
  3.  /etc/supervisord.conf
  4.  /etc/supervisor/supervisord.conf (since Supervisor 3.3.0)
  5.  ../etc/supervisord.conf (Relative to the executable)
  6.  ../supervisord.conf (Relative to the executable)

Below is the supervisord.conf i use for ADDS jar which i placed under /etc

logfile=/var/log/supervisord/supervisord.log    ; supervisord log file
logfile_maxbytes=50MB                           ; maximum size of logfile before rotation
logfile_backups=10                              ; number of backed up logfiles
loglevel=error                                  ; info, debug, warn, trace
pidfile=/var/run/                ; pidfile location
nodaemon=false                                  ; run supervisord as a daemon
minfds=1024                                     ; number of startup file descriptors
minprocs=200                                    ; number of process descriptors
user=root                                       ; default user
childlogdir=/var/log/supervisord/               ; where child log files will live
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
files = supervisor/conf.d/*.conf

App Config

in the ‘include’ section above, we want the supervisor to get all the conf files under conf.d. So now we can create a adds.conf there:

command=java -Dserver.port=8080 -Dlogging.path=/var/log/spring/adds/ -jar /var/adds-rest/adds-rest.jar

Run supervisord and its control

Now we can run ‘supervisord’ or sudo supervisord to start the daemon.

To fine control the processes, we can use the `supervisorctl` tool.Enter supervisorctl alone will take us the to interactive shell. and there are many actions we can take. use `help` to get the list.

So here we can do start/stop/restart/status…etc, a lot of stuff. 

Or we can run it directly using something like: `supervisorctl restart adds`. 

Web control console

We could access the supervisor control via http://IP:9001 and manage process there directly.