bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

install ora2pg on mac osx

Recently we need to migrate our on prem oracle database into postgresql on aws RDS. Looks like ora2pg is a nice little tool to mitigate the problem though not perfect.

I am running OS X 10.10.5 Yosemite. It already comes with perl v5.18.2 as well as cpan for installing packages.

download/install Oracle client

The official document specifies quite vaguely about what is the oracle client libraries.  Turns out we need to download 3 packages from oracle website, for my use case we have oracle 12c so my current version is 12.1.02. The packages you need are:

  1. Oracle Basic
  2. Sqlplus
  3. sdk

Then we unzip them to the SAME directory like: /Users/liha/develop/instantclient_12_1 .

Now we need to set 2 env variables for installation:

export LD_LIBRARY_PATH=/Users/liha/develop/instantclient_12_1
export ORACLE_HOME=/Users/liha/develop/instantclient_12_1

install DBI/DBD

Now we need to install DBI/DBD stuff that is required during the ora2pg Makefile phase, especially the DBD::Oracle. The DBD:Pg is optional, only if you need to dump data/schema live rather than offline to file.

#perl –MCPAN –e shell
…………
cpan> install DBI
…………
cpan> install DBD::Oracle
…………
cpan> install DBD::Pg
…………

Check module

Now we can check whether the dependencies are correctly installed by running a perl script.

 #!/usr/bin/perl
use ExtUtils::Installed;
my $inst = ExtUtils::Installed->new();
print join "\n",$inst->modules();

result should look like something:

➜  ora2pg perl test.pl                                                                                                                                                                              12:48:46
Bundle::NetSNMP
DBD::Oracle
DBI
Ora2Pg
Perl⏎

install ora2Pg

now can start to install the real thing by download and extract the ora2Pg tar.bz2 and run its makefile.pl which will eventually generate our makefile which we can make and install:

perl Makefile.PL
make && make install

After installation, the default conf file will be under etc/ora2pg/ora2pg.conf.

ora2Pg config

Now we can start config the ora2Pg by changing the default conf file or create a new conf file based on default conf. If you have multiple DB schemas to migrate or have multiple environments, it would be better to create different conf for each use case.

The important fields are:

ORACLE_DSN      dbi:Oracle:host=hostname;sid=Oracle SID

ORACLE_USER     schema_owner name

ORACLE_PWD      password

You should set as below to connect and retrieve required schema data and objects

USER_GRANTS     1

I also turned debug on to view progress

DEBUG           1

Other settings used were

EXPORT_SCHEMA   1

SCHEMA          schema_owner_name

CREATE_SCHEMA   1

I also used the same schema name in PG

PG_SCHEMA       schema_owner_name

I did have some problems with characters not being copied across as required and had to use settings below to override defaults, on some occasions

NLS_LANG       AMERICAN_AMERICA.UTF8

NLS_CHAR        LATIN1

Now you can run the dump:

ora2pg -c myOra2pg.conf -t TABLE -o tables.sql

另外一个中文文章

Install jdk 8 in Yosemite 10.10.2

While i was trying to downgrade JDK from 1.8.0._25 to 1.8.0_05 into the yosemite 10.10.2 so that i could use the DCEVM in my OSX.

 

I first remove the new version by just do a

sudo mv jdk1.8.0_25.jdk ~/Desktop/

Next step is to install the 1.8.0_05 which is the current latest version that DCEVM supports.However, I got the following error:

Your system has Mac OS X Version 10.10.2. This product can be installed on Version 10.7.3 or later.
Visit java.com/help for more information. 

Logically it looks really weird since apparently 10.10.2 is later than the 10.7.3. I think what they do is just compare the first digit so 1<7.

To install this, I have to spoof the OSX version.

1. go to  /System/Library/CoreServices and find the SystemVersion.plist which from its name we know contains the system version.

2. if you try to edit it directly you will find it r–r–r– readonly. To change it, you can either chmod it to 777 and modify content, or open it with some textEditor and do save as the same file name and extension. Then copy it back to the directory.

3. You can modify the version number to 9001 which is obviously larger than 10.10.1. I suggest you modify both ProductVersion and ProductUserVisibleVersion so that you could go to the “about this Mac” to verify the change.

4. Now you should be able to install without that error.

5. Do remember to change it back to whatever version you were before the modification.

 

.profile vs .bash_profile in OSX

According to the manual page that ships with OS X:

… it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

in summary: .profile works with sh or bash. However .bash_profile is bash-specific and will override .profile if present.

mac OS X environment checks .bash_profile, .bash_login, .profile in this order. It will run whichever is the highest in the hierarchy, so, if you have .bash_profile, it will not check .profile.

upgrade java version in OSX

To find the right value that JAVA_HOME should be set to.

1. first executed a `which java` in the nearest Terminal. this told me that /usr/bin/java was the executable that launched the VM.

2. A quick `ls -l /usr/bin/java` told me that it wasn’t a symbolic link to the JDK installation that I was hoping for. Instead, it is a short shell script that executes:
in the terminal, it’ll return /usr/bin/java. (which in turn points to /System/Library/Frameworks/JavaVM.framework/Versions/A/Commands/java, which is Apple’s 1.6 version).

3. So, if you want to use the new java version, replace the /usr/bin/java symlink so that it points to /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java instead.

4. The command could be:

sudo ln -fs /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java /usr/bin/java

5.Sanity check your version:

java -version

 

set up mongo db in windows and osx

OSX

1. download mongo and unzip to some folder, rename folder to mongodb. i put it under ~/Documents/develop/mongodb

2. create  folder “data” under “mongodb”.  create folder “db” under “data”

create folder “logs” under “mongodb”. create file “mongodb.log”

3. create file mongodb.conf under ~/Documents/develop/mongodb/bin with content:

dbpath = ~/Documents/develop/mongodb/data/db
directoryperdb = true
logpath = ~/Documents/develop/mongodb/logs/mongodb.log
logappend = true
rest = true

now you can run mongo db server under bin folder: “mongod -f mongodb.conf”

and start mongo by run “mongo” under bin folder.

to make it easier to start.

1. add a file “.profile” under ~/ with content:

# MongoDB server Alias
alias mongod=”`/Documents/develop/mongodb/bin/mongod –config=~/Documents/develop/mongodb/bin/mongod.conf”

#MongoDB instance Alias
alias mongo=”~/Documents/develop/mongodb/bin/mongo”

restart the shell, you should be able to run mongod and mongo directly .

Note: if you also have .bash_profile, then write the above content there since it has higher priority. More info HERE

Windows

step 1-3 is similar.

To make mongod a windows service, we can run:

C:\mongodb\bin\mongod.exe --config C:\mongodb\mongod.cfg --install

To run the MongoDB service:

net start MongoDB

Stop or Remove the MongoDB Service

To stop the MongoDB service:

net stop MongoDB

To remove the MongoDB service:

C:\mongodb\bin\mongod.exe --remove