run scp in background

recently we need to migrate our church’s Drupal site from godaddy vps to aws. The php/mysql part is kind of straightforward. The last task is to transfer the video/audio files to EBS. Since the size is pretty big(30~GB), we do not want to download to local and upload again. So scp becomes the choice. The command will be something like

scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files

But if the session terminates, it will be interrupted. So we need nohup to help.

nohup scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files

however this way, it runs in foreground not ideal. we need to add & to the end.

nohup scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files &

but now the problem is we have no way to input password and get the below error message:

nohup: ignoring input and appending output to `nohup.out'

this is just because the process is suspended to receive a user input for password.

Seems to be the dead end. Now the solution is still run the scp nohup in foreground so that we can input password, then use ctrl-z to suspend, and then use bg to send it to background. and if you want to bring it back, fg is the command to run. also use jobs command to check existing ones, or just use ps aux | grep scp.

Advertisements

base64 in shell

Used to go to some site when in need of en/de-code from/to base64. Turns out we can do that from the shell directly.

Encode

echo -n 'My TextTo Encode' | base64

The -n is to prevent the new line of the echo from being included in the result(by default echo outputs a trailing newline).

Decode

echo 'My Encoded TExt | base64 --decode.

we can also use -D(MacOS) or -d(Linux) instead of --decode for short.

clipboard in terminal(copy/paste)

Was watching a CD video with Semaphore and saw the guy in the video manipulating clipboard with pipe directly in the terminal. Pretty cool! Did some research on that.

OSX(MacOS)

In Mac, it is pretty straightforward with the built in pbcopy and pbpaste. (pasteboard copy / paste). Some examples:

cat my.key.pub | pbcopy

pbpaste > my.otherKey.pub

ps aux | grep myKeyWord | pbcopy

pbcopy < file.txt

pbpaste | grep myKeyWord

Linux

Take Ubuntu as example here. We can install a 16 kB program called xclip:

sudo apt-get install -y xclip

Then we can set alias so we can do the above operation like in osx. If you use bash then set your ~/.bashrc:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

If you are like me using fish as main shell, then similarly add below to your ~/.config/fish/config.fish file.

alias pbcopy "xclip -selection clipboard"
alias pbpaste "xclip -selection clipboard -o"

bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

install ora2pg on mac osx

Recently we need to migrate our on prem oracle database into postgresql on aws RDS. Looks like ora2pg is a nice little tool to mitigate the problem though not perfect.

I am running OS X 10.10.5 Yosemite. It already comes with perl v5.18.2 as well as cpan for installing packages.

download/install Oracle client

The official document specifies quite vaguely about what is the oracle client libraries.  Turns out we need to download 3 packages from oracle website, for my use case we have oracle 12c so my current version is 12.1.02. The packages you need are:

  1. Oracle Basic
  2. Sqlplus
  3. sdk

Then we unzip them to the SAME directory like: /Users/liha/develop/instantclient_12_1 .

Now we need to set 2 env variables for installation:

export LD_LIBRARY_PATH=/Users/liha/develop/instantclient_12_1
export ORACLE_HOME=/Users/liha/develop/instantclient_12_1

install DBI/DBD

Now we need to install DBI/DBD stuff that is required during the ora2pg Makefile phase, especially the DBD::Oracle. The DBD:Pg is optional, only if you need to dump data/schema live rather than offline to file.

#perl –MCPAN –e shell
…………
cpan> install DBI
…………
cpan> install DBD::Oracle
…………
cpan> install DBD::Pg
…………

Check module

Now we can check whether the dependencies are correctly installed by running a perl script.

 #!/usr/bin/perl
use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();

result should look like something:

➜  ora2pg perl test.pl                                                                                                                                                                              12:48:46
Bundle::NetSNMP
DBD::Oracle
DBI
Ora2Pg
Perl⏎

install ora2Pg

now can start to install the real thing by download and extract the ora2Pg tar.bz2 and run its makefile.pl which will eventually generate our makefile which we can make and install:

perl Makefile.PL
make && make install

After installation, the default conf file will be under etc/ora2pg/ora2pg.conf.

ora2Pg config

Now we can start config the ora2Pg by changing the default conf file or create a new conf file based on default conf. If you have multiple DB schemas to migrate or have multiple environments, it would be better to create different conf for each use case.

The important fields are:

ORACLE_DSN      dbi:Oracle:host=hostname;sid=Oracle SID

ORACLE_USER     schema_owner name

ORACLE_PWD      password

You should set as below to connect and retrieve required schema data and objects

USER_GRANTS     1

I also turned debug on to view progress

DEBUG           1

Other settings used were

EXPORT_SCHEMA   1

SCHEMA          schema_owner_name

CREATE_SCHEMA   1

I also used the same schema name in PG

PG_SCHEMA       schema_owner_name

I did have some problems with characters not being copied across as required and had to use settings below to override defaults, on some occasions

NLS_LANG       AMERICAN_AMERICA.UTF8

NLS_CHAR        LATIN1

Now you can run the dump:

ora2pg -c myOra2pg.conf -t TABLE -o tables.sql

另外一个中文文章

Shebang/Hashbang #! in shell

In unix sh file, we usually see the first line start with #!.

Like #!/bin/bash

The shebang (#!) at the start means execute the script with what follows.

In nodejs script, we usually see

#!/bin/env node

So this means the env command will try to find where the node has been installed from the $PATH, so that we do not have to specify the exact node path which would affect portability.