clipboard in terminal(copy/paste)

Was watching a CD video with Semaphore and saw the guy in the video manipulating clipboard with pipe directly in the terminal. Pretty cool! Did some research on that.

OSX(MacOS)

In Mac, it is pretty straightforward with the built in pbcopy and pbpaste. (pasteboard copy / paste). Some examples:

cat my.key.pub | pbcopy

pbpaste > my.otherKey.pub

ps aux | grep myKeyWord | pbcopy

pbcopy < file.txt

pbpaste | grep myKeyWord

Linux

Take Ubuntu as example here. We can install a 16 kB program called xclip:

sudo apt-get install -y xclip

Then we can set alias so we can do the above operation like in osx. If you use bash then set your ~/.bashrc:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

If you are like me using fish as main shell, then similarly add below to your ~/.config/fish/config.fish file.

alias pbcopy "xclip -selection clipboard"
alias pbpaste "xclip -selection clipboard -o"

bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

install ora2pg on mac osx

Recently we need to migrate our on prem oracle database into postgresql on aws RDS. Looks like ora2pg is a nice little tool to mitigate the problem though not perfect.

I am running OS X 10.10.5 Yosemite. It already comes with perl v5.18.2 as well as cpan for installing packages.

download/install Oracle client

The official document specifies quite vaguely about what is the oracle client libraries.  Turns out we need to download 3 packages from oracle website, for my use case we have oracle 12c so my current version is 12.1.02. The packages you need are:

  1. Oracle Basic
  2. Sqlplus
  3. sdk

Then we unzip them to the SAME directory like: /Users/liha/develop/instantclient_12_1 .

Now we need to set 2 env variables for installation:

export LD_LIBRARY_PATH=/Users/liha/develop/instantclient_12_1
export ORACLE_HOME=/Users/liha/develop/instantclient_12_1

install DBI/DBD

Now we need to install DBI/DBD stuff that is required during the ora2pg Makefile phase, especially the DBD::Oracle. The DBD:Pg is optional, only if you need to dump data/schema live rather than offline to file.

#perl –MCPAN –e shell
…………
cpan> install DBI
…………
cpan> install DBD::Oracle
…………
cpan> install DBD::Pg
…………

Check module

Now we can check whether the dependencies are correctly installed by running a perl script.

 #!/usr/bin/perl
use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();

result should look like something:

➜  ora2pg perl test.pl                                                                                                                                                                              12:48:46
Bundle::NetSNMP
DBD::Oracle
DBI
Ora2Pg
Perl⏎

install ora2Pg

now can start to install the real thing by download and extract the ora2Pg tar.bz2 and run its makefile.pl which will eventually generate our makefile which we can make and install:

perl Makefile.PL
make && make install

After installation, the default conf file will be under etc/ora2pg/ora2pg.conf.

ora2Pg config

Now we can start config the ora2Pg by changing the default conf file or create a new conf file based on default conf. If you have multiple DB schemas to migrate or have multiple environments, it would be better to create different conf for each use case.

The important fields are:

ORACLE_DSN      dbi:Oracle:host=hostname;sid=Oracle SID

ORACLE_USER     schema_owner name

ORACLE_PWD      password

You should set as below to connect and retrieve required schema data and objects

USER_GRANTS     1

I also turned debug on to view progress

DEBUG           1

Other settings used were

EXPORT_SCHEMA   1

SCHEMA          schema_owner_name

CREATE_SCHEMA   1

I also used the same schema name in PG

PG_SCHEMA       schema_owner_name

I did have some problems with characters not being copied across as required and had to use settings below to override defaults, on some occasions

NLS_LANG       AMERICAN_AMERICA.UTF8

NLS_CHAR        LATIN1

Now you can run the dump:

ora2pg -c myOra2pg.conf -t TABLE -o tables.sql

另外一个中文文章

Shebang/Hashbang #! in shell

In unix sh file, we usually see the first line start with #!.

Like #!/bin/bash

The shebang (#!) at the start means execute the script with what follows.

In nodejs script, we usually see

#!/bin/env node

So this means the env command will try to find where the node has been installed from the $PATH, so that we do not have to specify the exact node path which would affect portability.

 

Spring boot jar process management in AWS EC2 instance with supervisor

Purpose

Spring boot application is built into a jar which contains its own tomcat. So instead of running it in a traditional way of  having a tomcat instance that serving one or multiple wars, we will run the jar with java -jar. The problem here is if we run it directly, when our session quits/expires, the process will end. So we can either run it as a service(init.d), or using some 3rd party tool to manage it. In NodeJs world, we have the super powerful pm2. in the native world, ‘supervisor‘ seems to be the most recommended tool. Here we are going to introduce how to manage our spring boot jar with supervisor.

Install supervisor

Some EC2 AMI comes with easy_install’, which is a feature of setuptools. This is the preferred method of installation.

easy_install supervisor

 

Depending on the permissions of your system’s Python, you might need to be the root user to install Supervisor successfully using easy_install

Or we can use pip to install it thru: 

pip install supervisor

More reference in their official doc.

Supervisor Config

The Supervisor configuration file is conventionally named supervisord.conf. It is used by both supervisord and supervisorctl. If either application is started without the -coption (the option which is used to tell the application the configuration filename explicitly), the application will look for a file named supervisord.conf within the following locations, in the specified order. It will use the first file it finds.

  1.  $CWD/supervisord.conf
  2.  $CWD/etc/supervisord.conf
  3.  /etc/supervisord.conf
  4.  /etc/supervisor/supervisord.conf (since Supervisor 3.3.0)
  5.  ../etc/supervisord.conf (Relative to the executable)
  6.  ../supervisord.conf (Relative to the executable)

Below is the supervisord.conf i use for ADDS jar which i placed under /etc

[supervisord]
logfile=/var/log/supervisord/supervisord.log    ; supervisord log file
logfile_maxbytes=50MB                           ; maximum size of logfile before rotation
logfile_backups=10                              ; number of backed up logfiles
loglevel=error                                  ; info, debug, warn, trace
pidfile=/var/run/supervisord.pid                ; pidfile location
nodaemon=false                                  ; run supervisord as a daemon
minfds=1024                                     ; number of startup file descriptors
minprocs=200                                    ; number of process descriptors
user=root                                       ; default user
childlogdir=/var/log/supervisord/               ; where child log files will live
[inet_http_server]
port=9001
username=rc
password=rc
[supervisorctl]
username=rc
password=rc
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[include]
files = supervisor/conf.d/*.conf

App Config

in the ‘include’ section above, we want the supervisor to get all the conf files under conf.d. So now we can create a adds.conf there:

[program:adds]
command=java -Dserver.port=8080 -Dlogging.path=/var/log/spring/adds/ -jar /var/adds-rest/adds-rest.jar
user=app_fnotify_dev
autostart=true
autorestart=true
startsecs=10
startretries=3
stdout_logfile=/var/log/spring/adds/stdout.log
stderr_logfile=/var/log/spring/adds/stderr.log

Run supervisord and its control

Now we can run ‘supervisord’ or sudo supervisord to start the daemon.

To fine control the processes, we can use the `supervisorctl` tool.Enter supervisorctl alone will take us the to interactive shell. and there are many actions we can take. use `help` to get the list.

So here we can do start/stop/restart/status…etc, a lot of stuff. 

Or we can run it directly using something like: `supervisorctl restart adds`. 

Web control console

We could access the supervisor control via http://IP:9001 and manage process there directly.

OSX auto cleanup tmp ignore dir

Was putting some project related configs into the OS X /tmp directory, but found it was deleted. Turns out OS X has system scheduled cleanup job for the tmp dir. The config is under ‘/etc/defaults/periodic.conf‘.

I was first trying to leverage the provided ‘daily_clean_tmps_ignore’ parameter to exclude my directory but found not working, have to go to the source shell script to see why. The cleanup script locates in : ‘/etc/periodic/daily/110.clean-tmps’ which will go to the ‘/tmp’ dir and find command to get all the files that are 3 days old(atime, mtime, ctime) and not in the ignore list. So all the names we specified in the ‘daily_clean_tmps_ignore’ will be prefix the ‘! -name $pattern’ so that the $pattern could be excluded. However what i am trying to achieve here is for a directory and all its sub-files/dirs where using the -name is not enough.

We can use the -path option but we cannot add that thru configuration file.

Finally i decided to change the system script to achieve the ignoring by add a -name $pattern -prune option. After modification, it becomes:

 rc=$(for dir in $daily_clean_tmps_dirs
                do  
                    [ ."${dir#/}" != ."$dir" -a -d $dir ] && cd $dir && {
                        find -dx . -name 'MyDir' -prune -fstype local -type f $args -delete $print
                        find -dx . -name 'MyDir' -prune -fstype local ! -name . -type d $dargs -delete $print
                    } | sed "s,^\\.,  $dir,"
                done | tee /dev/stderr | wc -l)
            [ -z "$print" ] && rc=0
            [ $rc -gt 1 ] && rc=1
            set -f glob

A SO reference about -prune.