curl check url status

Recently we need to check our Route 53 accessibility before invalidating our cache in CDN. So need to add this logic into our Jenkins job.

The command :


r53_url="https://someURL/xx/index.html" 
status_code=$(curl --write-out %{http_code} --silent --output /dev/null ${cf_url})

if [[ "${status_code}" -ne 200 ]] ; then
  echo "Site status is ${status_code}. Make sure ${r53_url} is accessible first!" 
  exit 1
else
  echo "Connectivity to ${r53_url} verified. Invalidating Cache."
  ...
fi

Advertisements

work with multiple files in vim

When working with multi files in vim, there are several options.

Netrw

Was first trying to use the netrw which comes with vim installation. However it is not quite easy to use. :e. is to open the file explore, or :Exp, or : Vex, or :Sex.

NERDTree

Nerdtree is the de facto file explore. After the frustration with netrw, i decided to give it a a try and it has so many github stars for some reasons. Unlike a lot of custom config in netrw, most nerdtree config is out of box.

  • s to open a file in a file in a vertical spli
  • I to show hidden files in explorer
  • m to open menu with options to do CRUD.
  • Tabs also works nice with t open in new tab and gt or #NUMBERgt to navigate between them. and use ctrl + w to toggle focus.
  • ? to show the available shortcuts.

Vim plug

vim-plug is a new vim plugin management tool which seems to be better than others like vbundle. it allows lazy loading plugins which is awesome. The config is also easy, just add the desired plugin into .vimrc and run :PlugInstall. To remove plugin, delete the line in .vimrc and run :PluginClear.

super tab

super tab is nice plugin for word/code completion. it leverages the vim popup for completion and the index is also smart.

terminal in vim

if you have vim 8, then :term is built in. (`:echo has(‘terminal’)` will output “1”.). Same as tabs in NERDTree, use `ctrl+w` to navigate between terminal and other tabs.

in macos use brew install vim --override-system-vi to upgrade to vim 8.

run scp in background

recently we need to migrate our church’s Drupal site from godaddy vps to aws. The php/mysql part is kind of straightforward. The last task is to transfer the video/audio files to EBS. Since the size is pretty big(30~GB), we do not want to download to local and upload again. So scp becomes the choice. The command will be something like

scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files

But if the session terminates, it will be interrupted. So we need nohup to help.

nohup scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files

however this way, it runs in foreground not ideal. we need to add & to the end.

nohup scp -r vpsUser@vpsDomain:/vps/site/path/files/audio /aws/ebs/files &

but now the problem is we have no way to input password and get the below error message:

nohup: ignoring input and appending output to `nohup.out'

this is just because the process is suspended to receive a user input for password.

Seems to be the dead end. Now the solution is still run the scp nohup in foreground so that we can input password, then use ctrl-z to suspend, and then use bg to send it to background. and if you want to bring it back, fg is the command to run. also use jobs command to check existing ones, or just use ps aux | grep scp.

base64 in shell

Used to go to some site when in need of en/de-code from/to base64. Turns out we can do that from the shell directly.

Encode

echo -n 'My TextTo Encode' | base64

The -n is to prevent the new line of the echo from being included in the result(by default echo outputs a trailing newline).

Decode

echo 'My Encoded TExt | base64 --decode.

we can also use -D(MacOS) or -d(Linux) instead of --decode for short.

clipboard in terminal(copy/paste)

Was watching a CD video with Semaphore and saw the guy in the video manipulating clipboard with pipe directly in the terminal. Pretty cool! Did some research on that.

OSX(MacOS)

In Mac, it is pretty straightforward with the built in pbcopy and pbpaste. (pasteboard copy / paste). Some examples:

cat my.key.pub | pbcopy

pbpaste > my.otherKey.pub

ps aux | grep myKeyWord | pbcopy

pbcopy < file.txt

pbpaste | grep myKeyWord

Linux

Take Ubuntu as example here. We can install a 16 kB program called xclip:

sudo apt-get install -y xclip

Then we can set alias so we can do the above operation like in osx. If you use bash then set your ~/.bashrc:

alias pbcopy='xclip -selection clipboard'
alias pbpaste='xclip -selection clipboard -o'

If you are like me using fish as main shell, then similarly add below to your ~/.config/fish/config.fish file.

alias pbcopy "xclip -selection clipboard"
alias pbpaste "xclip -selection clipboard -o"

bitten by system case sensitivity twice today

Today i was bitten by system case insensitivity twice:

  1. Maven test does not pick up Spring boot test profile properties
  2. git commit does not get the file name change.

Spring boot profile based config

In spring boot, we can have different resource files(.properties or .yml ) to hold properties. More over, we could have profile specific ones, like application-dev.yml will be loaded if dev is in the ActiveProfiles.

In our project, I passed in TEST as profile when doing junit test and the resource file name happens to be application-test.yml. Things work prefect in my local(OSX) as well as my colleagues(Windows). However when we do our CI build in a Docker container which has CentOS, I get exceptions for database url which is environment specific is not loaded. Spent more than 1 hour working with our CM to figure this out. So I changed test profile name to test and it worked.

git commit ignore file name case change

I was trying to change a class name from ReportdaoTest to ReportDaoTest but after check in, CI build failed again since only the class name changed but the file name remains the same.

GIT is case-sensitive but some file systems, like HFS, are not. Git can’t detect your changes, because the file names didn’t change really.

The two most popular operating systems that have case-insensitive file systems that I know of are

  • Windows
  • OS X

So to fix this, we can do either:

  1. git mv -f OldFileNameCase newfilenamecase
  2.  git config –global core.ignorecase false

For this SO question.

summary

A good lesson. Previously never thought about this could cause trouble. I was thing it was char encoding problem and turns out not.