bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.

cli for postgres in linux

If  your PostgreSQL server is not started, start it with the following command.

sudo service postgresql start

To getting started with postgreSQL, open your Terminal, and type:

username$ sudo su - postgres

Now you are logged into your vm with the postgres user. Then connect to template1 database using the postgresql’s psql client:

postgres ~$  psql template1
postgres ~$  template1=#

Note the # symbol, declaring that you are logged to the database as a Super User.

Once you are done and want to disconnect from the database using \q, and quit from postgres user using ctrl+d.

Using Psql client

Let’s see some useful commands of the psql tool. First of all: you can get some help just typing \?in your psql command line. To exit from the appearing screen, just press the q button.

To load and execute SQL statemens from file, you can use this syntax:

\i my/file/path.sql

Writing long lines of SQL code in the command line can be really hard to debug. Use the \i to run from file or use:

\e

To edit the input buffer with an editor. If you usually don’t use editor from command line use nano – it’s the simplest. To check what relations your database contains, use

\d

If you want more information about a particular relation, use

\d myfancytable

Have fun!

MY unix commands

tail a file

tail -100f FileName

-f to allow file monitoring

less command

Another way is to watch file using less +F FileName to wait for new data. use ctrl+c to exit to the normal less mode to do search/page uptown etc. once down, we can hit F again to the watch mode.

less only opens chunk of file so should be more memory friendly than vim when opening large file. And its command set is very similar to vi. The con is the syntax highlight is not that good comparing to vim.

check process

ps -f -p ThePID

-f full all columns

-p by process ID

ps -e -f

-e all processes

ps -ef | grep yourkeyword

to search for process

if the CMD is very long like running java command with long classpath and param. use

-ww

or

ww

Wide output. Use this option twice for unlimited width.

Extract/Compress file

tar -xvf YOUR.tar.gz

xf are required: The -x (for extract). The -f indicates that the next argument (i.e., piece of input data in a command) will be the name of the new archive file.

The -v (i.e., verbose) option is commonly used together with the -c/-x and -f options in order to display a list of the files that are included in the archive.

-c instructs tar to create an archive:

tar -cvf file.tar file1 file2 file3   OR  tar -cf * to compress all.

The -t option tells tar to list the contents of an uncompressed archive without performing an extraction. Thus, the following would list the contents of file.tar:

tar -tf file.tar

More reference like delete/add file to tar.

remove file/direcotry

rm to remove file. To remove non-empty directory:

rm -rf dirToBeRemoved/
  1. -r : Attempt to remove the file hierarchy rooted in each file argument i.e. recursively remove subdirectories and files from the specified directory.
  2. -f : Attempt to remove the files without prompting for confirmation, regardless of the file’s permissions

disk usage

df -h

-h: human readable display

check directory/fold size

du -sh folderName

with out ‘s’, it would print out size recursively

to see all the directory and sort by size reversely:

du -sh * | sort -nr

find difference for two files

Option 1: diff -y file1 file2

-y will display output in two columns. More options about diff

-i is also handy when we want to ignore case

To get only diff, we can add a pipe and grep lines with ‘|’ like:

diff -y file1 file2 | grep -E ".*(\||>).*"

-E in grep means: Interpret PATTERN as an extended regular expression

Option2: grep -vf file1 file2

-v Invert the sense of matching, to select non-matching lines.

-f Obtain patterns from FILE, one per line.

so the above command will output the lines in file2 but not in file1. And we can run the reverse: grep -vf file2 file1 to get the lines in file1 but not in file2.

The first param could also be just a String  like: ‘grep -v apple fruitList.txt’ will get all lines in fruitList.text that is not apple.

grep man page

find string in current directory

grep -nr yourString* .

The dot at the end searches the current directory. Meaning for each parameter:

'-n'          Show relative line number in the file
'yourString*' String for search
'-r'          Recursively search subdirectories listed
'.'           Directory for search (current directory)

find code line number

find . -type f -name "*.java" -or -name "*.xml" | xargs cat | wc -l

list who is listening a port

lsof -i :PORT_NUMBER

OR

netstat -anp | grep PORT_NUMBER

OR

fuser PORT_NUMBER/tcp

and to kill these processes, just add -k: fuser -k 10000/tcp

Shutdown and reboot

use ‘halt’ and ‘reboot’ to do shutdown/reboot in cmd.

 replace all in VI

:%s/search_string/replacement_string/g

verfiy http/https connectivity

HERE

For general ip/port connectivity test, use nc(netcat)

nc -zv YourIp YourPort

-z : Specifies that nc should just scan for listening daemons, without sending any data to them. It is an error to use this option in conjunction with the -l option.

-v: verbose

change permissions

HERE

create link(ln) to file/directory

HERE

about sudo/su

HERE

about make file immutable

With chattr, we can make a file immutable or append-only. We can use lsattr to view the attritbutes.

HERE

linux/unix SHELL 是什么

shell是你(用户)和Linux(或者更准确的说,是你和Linux内核)之间的接口程序。你在提示符下输入的每个命令都由shell先解释然后传给Linux内核。
——————————————————————————–

注意: 如果你熟悉ms-dos,你将认出这很象DOS 用户和 COMMAND.COM之间的关系。唯
一的区别是command.com 的功能远不能和shell 相提并论。

——————————————————————————–

shell 是一个命令语言解释器(command-language interpreter)。拥有自己内建的 shell 命令集。此外,shell也能被系统中其他有效的Linux 实用程序和应用程序(utilities and application programs)所调用。

不论何时你键入一个命令,它都被Linux shell所解释。一些命令,比如打印当前工作目录命令(pwd),是包含在Linux bash内部的(就象DOS的内部命令)。其他命令,比如拷贝命令(cp)和移动命令(rm),是存在于文件系统中某个目录下的单独的程序。而对用户来说,你不知道(或者可能不关心)一个命令是建立在shell内部还是一个单独的程序。

附图展示了 shell 是如何执行命令解释的,shell 首先检查命令是否是内部命令,不是的话再检查是否是一个应用程序,这里的应用程序可以是Linux本身的实用程序,比如ls 和 rm,也可以是购买的商业程序,比如 xv,或者是公用软件(public domain software),就象 ghostview。然后shell试着在搜索路径里寻找这些应用程序。搜索路径是一个能找到可执行程序的目录列表。如果你键入的命令不是一个内部命令并且在路径里没有找到这个可执行文件,将会显示一条错误信息。而如果命令被成功的找到的话,shell的内部命令或应用程序将被分解为系统调用并传给Linux内核。

shell的另一个重要特性是它自身就是一个解释型的程序设计语言,shell 程序设计语言支持在高级语言里所能见到的绝大多数程序控制结构,比如循环,函数,变量和数组。shell 编程语言很易学,并且一旦掌握后它将成为你的得力工具。任何在提示符下能键入的命令也能放到一个可执行的shell程序里,这意味着用shell语言能简单地重复执行某一任务。

 

shell 是一个交互性命令解释器。shell独立于操作系统,这种设计让用户可以灵活选择适合自己的shell。shell让你在命令行键入命令,经过shell解释后传送给操作系统(内核)执行。
    shell是一个命令处理器(command processor)——是一个读入并解释你输入的命令的程序。除了是一个命令中断器以外,shell还是一个程序设计语言。你可以编写shell可以解释的程序(被称为源程序),这些源程序可以包含shell程序设计命令等等。shell除了解释命令以外,还有其他工作,它也可以配置和编程。   
    shell拥有自己的语言允许用户编写程序并以一种复杂方式运行。shell编程语言具有许多常用的编程语言的特征,例如:循环和控制结构等。用户可以生成像其他应用程序一样复杂的shell程序。
以下是shell功能的一个汇总:
查找命令的位置并且执行相关联的程序;
为shell变量赋新值;
执行命令替代;
处理 I/O重定向和管道功能;
提供一个解释性的编程语言界面,包括tests、branches和loops等语句。

    bash是borne again shell的缩写,它是shell的一种,Linux上默认采用的是bash。
    当你在命令行中敲入bash命令时,相当于进入bash环境,如果本身就是bash环境,那么就是进入一个子bash环境(相当于开了一个子进程)。

 

FROM HERE

JAVA_HOME in windows 7

Windows:

  • The SetX command would do so:

    setx /M JAVA_HOME “C:\Program Files (x86)\Java\jdk1.6.0_17”

    /m means machine level.

  • To check the settings, use “set” command for all the variables.

    Use “echo %JAVA_HOME%” to check it specifically.

  • To append to the PATH:

    setx path “%PATH%;c:\home” /m

Here is a good site for windows commands 

 

Unix

In Unix, go to “/etc/profile” and add “export PATH=$PATH:/usr/java/jdk1.5.0_07/bin”.