MY unix commands

tail a file

tail -100f FileName

-f to allow file monitoring

less command

Another way is to watch file using less +F FileName to wait for new data. use ctrl+c to exit to the normal less mode to do search/page uptown etc. once down, we can hit F again to the watch mode.

less only opens chunk of file so should be more memory friendly than vim when opening large file. And its command set is very similar to vi. The con is the syntax highlight is not that good comparing to vim.

check process

ps -f -p ThePID

-f full all columns

-p by process ID

ps -e -f

-e all processes

ps -ef | grep yourkeyword

to search for process

if the CMD is very long like running java command with long classpath and param. use

-ww

or

ww

Wide output. Use this option twice for unlimited width.

Extract/Compress file

tar -xvf YOUR.tar.gz

xf are required: The -x (for extract). The -f indicates that the next argument (i.e., piece of input data in a command) will be the name of the new archive file.

The -v (i.e., verbose) option is commonly used together with the -c/-x and -f options in order to display a list of the files that are included in the archive.

-c instructs tar to create an archive:

tar -cvf file.tar file1 file2 file3   OR  tar -cf * to compress all.

The -t option tells tar to list the contents of an uncompressed archive without performing an extraction. Thus, the following would list the contents of file.tar:

tar -tf file.tar

More reference like delete/add file to tar.

remove file/direcotry

rm to remove file. To remove non-empty directory:

rm -rf dirToBeRemoved/
  1. -r : Attempt to remove the file hierarchy rooted in each file argument i.e. recursively remove subdirectories and files from the specified directory.
  2. -f : Attempt to remove the files without prompting for confirmation, regardless of the file’s permissions

disk usage

df -h

-h: human readable display

check directory/fold size

du -sh folderName

with out ‘s’, it would print out size recursively

find difference for two files

Option 1: diff -y file1 file2

-y will display output in two columns. More options about diff

-i is also handy when we want to ignore case

To get only diff, we can add a pipe and grep lines with ‘|’ like:

diff -y file1 file2 | grep -E ".*(\||>).*"

-E in grep means: Interpret PATTERN as an extended regular expression

Option2: grep -vf file1 file2

-v Invert the sense of matching, to select non-matching lines.

-f Obtain patterns from FILE, one per line.

so the above command will output the lines in file2 but not in file1. And we can run the reverse: grep -vf file2 file1 to get the lines in file1 but not in file2.

The first param could also be just a String  like: ‘grep -v apple fruitList.txt’ will get all lines in fruitList.text that is not apple.

grep man page

find string in current directory

grep -nr yourString* .

The dot at the end searches the current directory. Meaning for each parameter:

'-n'          Show relative line number in the file
'yourString*' String for search
'-r'          Recursively search subdirectories listed
'.'           Directory for search (current directory)

find code line number

find . -type f -name "*.java" -or -name "*.xml" | xargs cat | wc -l

list who is listening a port

lsof -i :PORT_NUMBER

OR

netstate -anp | grep PORT_NUMBER

Shutdown and reboot

use ‘halt’ and ‘reboot’ to do shutdown/reboot in cmd.

 replace all in VI

:%s/search_string/replacement_string/g

verfiy http/https connectivity

HERE

For general ip/port connectivity test, use nc(netcat)

nc -zv YourIp YourPort

-z : Specifies that nc should just scan for listening daemons, without sending any data to them. It is an error to use this option in conjunction with the -l option.

-v: verbose

change permissions

HERE

create link(ln) to file/directory

HERE

about sudo/su

HERE

about make file immutable

With chattr, we can make a file immutable or append-only. We can use lsattr to view the attritbutes.

HERE

linux/unix SHELL 是什么

shell是你(用户)和Linux(或者更准确的说,是你和Linux内核)之间的接口程序。你在提示符下输入的每个命令都由shell先解释然后传给Linux内核。
——————————————————————————–

注意: 如果你熟悉ms-dos,你将认出这很象DOS 用户和 COMMAND.COM之间的关系。唯
一的区别是command.com 的功能远不能和shell 相提并论。

——————————————————————————–

shell 是一个命令语言解释器(command-language interpreter)。拥有自己内建的 shell 命令集。此外,shell也能被系统中其他有效的Linux 实用程序和应用程序(utilities and application programs)所调用。

不论何时你键入一个命令,它都被Linux shell所解释。一些命令,比如打印当前工作目录命令(pwd),是包含在Linux bash内部的(就象DOS的内部命令)。其他命令,比如拷贝命令(cp)和移动命令(rm),是存在于文件系统中某个目录下的单独的程序。而对用户来说,你不知道(或者可能不关心)一个命令是建立在shell内部还是一个单独的程序。

附图展示了 shell 是如何执行命令解释的,shell 首先检查命令是否是内部命令,不是的话再检查是否是一个应用程序,这里的应用程序可以是Linux本身的实用程序,比如ls 和 rm,也可以是购买的商业程序,比如 xv,或者是公用软件(public domain software),就象 ghostview。然后shell试着在搜索路径里寻找这些应用程序。搜索路径是一个能找到可执行程序的目录列表。如果你键入的命令不是一个内部命令并且在路径里没有找到这个可执行文件,将会显示一条错误信息。而如果命令被成功的找到的话,shell的内部命令或应用程序将被分解为系统调用并传给Linux内核。

shell的另一个重要特性是它自身就是一个解释型的程序设计语言,shell 程序设计语言支持在高级语言里所能见到的绝大多数程序控制结构,比如循环,函数,变量和数组。shell 编程语言很易学,并且一旦掌握后它将成为你的得力工具。任何在提示符下能键入的命令也能放到一个可执行的shell程序里,这意味着用shell语言能简单地重复执行某一任务。

 

shell 是一个交互性命令解释器。shell独立于操作系统,这种设计让用户可以灵活选择适合自己的shell。shell让你在命令行键入命令,经过shell解释后传送给操作系统(内核)执行。
    shell是一个命令处理器(command processor)——是一个读入并解释你输入的命令的程序。除了是一个命令中断器以外,shell还是一个程序设计语言。你可以编写shell可以解释的程序(被称为源程序),这些源程序可以包含shell程序设计命令等等。shell除了解释命令以外,还有其他工作,它也可以配置和编程。   
    shell拥有自己的语言允许用户编写程序并以一种复杂方式运行。shell编程语言具有许多常用的编程语言的特征,例如:循环和控制结构等。用户可以生成像其他应用程序一样复杂的shell程序。
以下是shell功能的一个汇总:
查找命令的位置并且执行相关联的程序;
为shell变量赋新值;
执行命令替代;
处理 I/O重定向和管道功能;
提供一个解释性的编程语言界面,包括tests、branches和loops等语句。

    bash是borne again shell的缩写,它是shell的一种,Linux上默认采用的是bash。
    当你在命令行中敲入bash命令时,相当于进入bash环境,如果本身就是bash环境,那么就是进入一个子bash环境(相当于开了一个子进程)。

 

FROM HERE

$# and S* in shell script

From here:

$#    Stores the number of command-line arguments that 
      were passed to the shell program.
$?    Stores the exit value of the last command that was 
      executed.
$0    Stores the first word of the entered command (the 
      name of the shell program).
$*    Stores all the arguments that were entered on the
      command line ($1 $2 ...).
"$@"  Stores all the arguments that were entered
      on the command line, individually quoted ("$1" "$2" ...).

So basically, $# is a number of arguments given when your script was executed. $* is a string containing all arguments. For example, $1 is the first argument and so on. This is useful, if you want to access a specific argument in your script.

As Brian commented, here is a simple example. If you run following command:

./command -yes -no /home/username
  • $# = 3
  • $* = -yes -no /home/username
  • $@ = array: {"-yes", "-no", "/home/username"}
  • $0 = ./command, $1 = -yes etc.

FROM HERE

中文:

$0 程序名字
$n 第n个参数值,n=1..9
$* 所有命令行参数
$@ 所有命令行参数,如果它被包含在引号里,形如”$@”,则每个参数也各自被引号包括
$# 命令行参数个数
$$ 当前进程的进程ID(PID)
$!  最近后台进程的进程ID
$?  最近使用命令的退出状态

Ubuntu进入root权限命令——sudo和su命令详解

ubuntu用户管理

sudo 意思就是super-user do,让当前用户暂时以管理员的身份root来执行这条命令。

su (switch user)是用来改变当前用户的,su root,就是将当前用户切换为root,用了su root之后,下面所有的命令就可以不用打sudo了,因为当前用户已经是管理员root了。

root 用户为根用户,也就是 系统管理员 拥有全部权限

一个用户只能拥有一个 GID ,但是还可以归属于其它附加群组

 

用户管理的重要配置文件:

/etc/passwd 用户名 密码位 UID 归属GID 姓名 $HOME目录 登录Shell/etc/shadow 用户名 已加密密码 密码改动信息 密码策略/etc/group 群组名 密码位 GID 组内用户/etc/gshadow 群组密码相关文件,不重要/etc/sudoers 用户名 权限定义 权限[/pre]可以使用 pwconv 命令创建影子密码,将 /etc/passwd 文件中的密码转换到 /etc/shadow 文件

 

su [用户名]

切换到其它用户,默认切换到root用户。提示密码为将切换用户密码 -f 快速切换,忽略配置文件-l 重新登录-m -p 不更改环境变量-c <命令> 切换后执行命令,并退出切换

 

sudo [命令]

以其它用户的身份执行命令,默认以root的身份执行。提示密码为当前用户密码 -s 切换为root shell-i 切换为root shell,并初始化-u <用户名|UID> 执行命令的身份-l 显示自己的权限

 

passwd [用户名]

设定用户密码

-d 清除密码 -l 锁定用户-e 使密码过期,在下次登录时更改密码-S 显示密码认证信息-x <天数> 密码过期,最大使用时间-n <天数> 冻结密码,最小使用时间-s 更改登录Shell-f 更改用户信息

示例:

$passwdChanging password for user(current) UNIX password: 原密码Enter new UNIX password: 新密码Retype new UNIX password: 确认新密码[/pre

 

chsh [-s ] [用户名]

更改登录Shell

1.sudo的缺省配置

默认时,Ubuntu sudo提供了一个基本的配置,该配置保存在/etc目录下的sudoers文件中。在修改该配置文件时,务必使用visudo工具来进行编辑,因为该工具会自动对配置语法进行严格检查,如果发现错误,在保存退出时给出警告,并提示你哪段配置出错,从而确保该配置文件的正确性。相反, 如果使用其它的文本编辑程序的话,一旦出错,就会给系统带来严重的后果。下面给出的是Ubuntu默认的/etc/sudoers文件内容:

# User privilege specification

root ALL=(ALL) ALL

# Members of the admin group may gain root privileges

admin ALL=(ALL) ALL

下面对以上配置做简要说明:

第 一项配置的作用,是允许root用户使用sudo命令变成系统中任何其它类型的用户。

第二个配置规定,管理组中的所有成员都能以root的身份执行所有命令。

因此,在默认安装的Ubuntu系统中,要想作为root身份来执行命令的话,只要在sudo后面跟上欲执行的命令即可。

下面用一个例子加以说明,如果您想执行apt-get update的话,应当在命令行中键入以下内容:

$ sudo apt-get update

2.配置文件语法详解

接下来,我们用一个实例来详细解释/etc/sudoers文件的配置语法,请看下面的例子:

jorge ALL=(root) /usr/bin/find, /bin/rm

上面的第一栏规定它的适用对象:用户或组,就本例来说,它是用户jorge。此外,

因为系统中的组和用户可以重名,要想指定该规则的适用对象是组而非用户的话,组对象的名称一定要用百分号%开头。

第二栏指定该规则的适用主机。当我们在多个系统之间部署sudo环境时,这一栏格外有用,这里的ALL代表所有主机。但是,对于桌面系统或不想将 sudo部署到多个系统的情况,这一栏就换成相应的主机名。

第三栏的值放在括号内,指出第一栏规定的用户能够以何种身份来执行命令。本例中该值设为root,这意味着用户jorge能够以root用户的身份 来运行后面列出的命令。该值也可以设成通配符ALLjorge便能作为系统中的任何用户来执行列出的命令了。

最后一栏(即/usr/bin/find, /bin/rm)是使用逗号分开的命令表,这些命令能被第一栏规定的用户以第三栏指出的身份来运行它们。本例中,该配置允许jorge作为超级用户运行 /usr/bin/find /bin/rm这两个命令。需要指出的是,这里列出的命令一定要使用绝对路径。

3.sudo命令的使用方法

现在的问题是,用户jorge怎样利用分配给他的权限呢?

其实很简单,只要在命令行模式下使用sudo命令

加上他想运行的程序就可以了,比如:

jorge@Ubuntu:~$ sudo find . ! -name ‘*.avi’ -exec rm -f //{//} //;

倘若jorge企图执行/etc/sudoers文件规定之外的程序(比如find rm)的话,sudo命令便会以失败而告终,并给出警告信息,指出他无权以超级用户身份来运行这些命令。

要想以非root用户身份来运行命令,必须使用-u选项来指定想要作为的用户;否则的话,sudo会默认为root用户,比如要想以fred身份来 执行ls命令,就应该这样:

$ sudo -u fred ls /home/fred

就 像您看到的那样,我们可以利用这些规则为系统创建具体的角色。例如,要让一个组负责帐户管理,你一方面不想让这些用户具备完全的root访问权限,另一方 面还得让他们具有增加和删除用户的权利,那么我们可以在系统上创建一个名为accounts的组,然后把那些用户添加到这个组里。之后,再使用 visudo/etc/sudoers添加下列内容:accounts ALL=(root) /usr/sbin/useradd, /usr/sbin/userdel, /usr/sbin/usermod

现在好了,accounts组中的任何成员都能运行useradduserdelusermod命令了。如果过一段时间后,您发现该角色还需要其他工具,只要在该表的尾部将其添上就行了。这样真是方便极了!

需要注意的是,当我们为用户定义可以运行的命令时,必须使用完整的命令路径。这样做是完全出于安全的考虑,如果我们给出的命令只是简单的userad而非 /usr/sbin/useradd,那么用户有可能创建一个他自己的脚本,也叫做userad,然后放在它的本地路径中,如此一来他就能够通过这个名为 useradd的本地脚本,作为root来执行任何他想要的命令了。这是相当危险的!

sudo命令的另一个便捷的功能,是它能够指出哪些 命令在执行时不需要输入密码。这很有用,尤其是在非交互式脚本中以超级用户的身份来运行某些命令的时候。例如,想要让用户作为超级用户不必输入密码就能执 行kill命令,以便用户能立刻杀死一个失控的进程。为此,在命令行前边加上NOPASSWD:属性即可。例如,可以在/etc/sudoers文件中加 上下面一行,从而让jorge获得这种权力:jorge ALL=(root) NOPASSWD: /bin/kill, /usr/bin/killall

这样一来,jorge就能运行以下命令,作为root用户来杀死失控的rm进程了。jorge@Ubuntu:~$ sudo killall rm

6.如何启用root帐户

通过以上介绍,我们发现sudo的确很好用,但是如果您早就习惯了在root下工作,想回味一下过去的感觉该怎么办呢?很简单,只要为root设置 一个root密码就行了:$ sudo passwd root

好了,现在您能直接作为root登录了

unix privs chmod user group Permissions

Introduction

This is a topic that has been beaten to death both in books and on-line. For some reason, it seems that it is one of the most common misunderstandings that people have to face when learning how to write and/or configure their first cgi programs. This tutorial aims to clarify the concepts involved. Note that we will be referring to UNIX in a generic sense in this article. Most of what we are going to discuss here applies to all UNIX flavours. (such as Linux, SVR4, BSD etc.) It is also a good idea to type man chmod to check for the specific details on your system, too.

Users

A UNIX system serves many users. Users are an abstraction that denotes a logical entity for assignment of ownership and operation privileges over the system. A user may correspond to a real-world person, but also a type of system operation. So, in my system, I have user ‘nick’ that corresponds to me, but I also have user ‘www’ which corresponds to the privileges necessary to operate the local webserver. UNIX doesn’t care about what the user means for me. It just knows what belongs to any given user and what each user is allowed to do with any given thing (file, program, device, etc) on the system. UNIX identifies each user by a User ID (UID) and the username (or login) such as ‘nick’ and ‘www’ is just an alias to the UID that makes humans more comfortable.

 

Groups

 

Users can be organized in groups. A user may belong to one or more groups of users. The concept of groups serves the purpose of assigning sets of privileges for a given resource and sharing them among many users that need to have them. (perhaps because they are all members of a project working team and they all need access to some common project files) So, on my system user ‘nick’ and user ‘www’ both belong to the group ‘perlfect’. This way, they can have some shared privileges over the files for this site. User ‘nick’ needs them to edit the site, and user ‘www’ needs them to manage the webserver that will be publishing the site.

 

Ownership

 

Every file in UNIX has an owner user and an owner group. So, for any file in the system, user ‘nick’ may have one of the following ownership relations:

  • nick owns the file, i.e. the file’s owner is ‘nick’.
  • nick is a member of the group that owns the file, i.e. the file’s owner group is ‘perlfect’.
  • nick is neither the owner, nor belonging to the group that owns the file

 

Permissions

 

Every file on the system has associated with it a set of permissions. Permissions tell UNIX what can be done with that file and by whom. There are three things you can (or can’t) do with a given file:

  • read it,
  • write (modify) it and
  • execute it.

Unix permissions specify which of the above operations can be performed for any ownership relation with respect to the file. In simpler terms, what can the owner do, what can the owner group do, and what can everybody else do with the file. For any given ownership relation, we need three bits to specify access permissions: the first to denote read (r) access, the second to denote (w) access and the third to denote execute (x) access. We have three ownership relations: ‘owner’, ‘group’ and ‘all’ so we need a triplet for each, resulting in nine bits. Each bit can be set or clear. (not set) We mark a set bit by it’s corresponding operation letter (r, w or x) and a clear bit by a dash (-) and put them all on a row. An example might be rwxr-xr-x.What this means is that the owner can do anything with the file, but group owners and the rest of the world can only read or execute it. Usually in UNIX there is also another bit that precedes this 9-bit pattern. You do not need to know about it, at least for the time being.

 

So if you try ls -l on the command prompt you will get something like the following:

[nick@thekla src]$ ls -l
-rwxr-xr-x   1 nick     users          382 Jan 19 11:49 bscoped.pl
drwxr-xr-x   3 nick     users         1024 Jan 19 11:19 lib/
-rwxr-xr-x   1 nick     users         1874 Jan 19 10:23 socktest.pl

 

The first column here shows the permission bit pattern for each file. The third column shows the owner, and the fourth column shows the owner group. By the time, the information provided by ls -l should be enough for you to figure out what each user of the system can do with any of the files in the directory.

 

Directories

 

Another interesting thing to note is that lib/ which is a directory has permissions, too. Permissions take a different meaning for directories. Here’s what they mean:

  • read determines if a user can view the directory’s contents, i.e. do ls in it.
  • write determines if a user can create new files or delete file in the directory. (Note here that this essentially means that a user with write access toa directory can delete files in the directory even if he/she doesn’t have write permissions for the file! So be careful with this.)
  • execute determines if the user can cd into the directory.

 

chmod

 

To set/modify a file’s permissions you need to use the chmod program. Of course, only the owner of a file may use chmod to alter a file’s permissions. chmod has the following syntax: chmod [options] mode file(s)

 

The ‘mode’ part specifies the new permissions for the file(s) that follow as arguments. A mode specifies which user’s permissions should be changed, and afterwards which access types should be changed. Let’s say for example:

chmod a-x socktest.pl

This means that the execute bit should be cleared (-) for all users. (owner, group and the rest of the world) The permissions start with a letter specifying what users should be affected by the change, this might be any of the following:

  • u the owner user
  • g the owner group
  • o others (neither u, nor g)
  • a all users

This is followed by a change instruction which consists of a +(set bit) or (clear bit) and the letter corresponding to the bit that should be changed.

 

Let’s see some examples:

$ ls -l socktest.pl 
-rwxr-xr-x   1 nick     users         1874 Jan 19 10:23 socktest.pl*

$ chmod a-x socktest.pl 
$ ls -l socktest.pl 
-rw-r--r--   1 nick     users         1874 Jan 19 10:23 socktest.pl

$ chmod g+w socktest.pl 
$ ls -l socktest.pl 
-rw-rw-r--   1 nick     users         1874 Jan 19 10:23 socktest.pl

$ chmod ug+x socktest.pl 
$ ls -l socktest.pl 
-rwxrwxr--   1 nick     users         1874 Jan 19 10:23 socktest.pl*

$ chmod ug-wx socktest.pl 
$ ls -l socktest.pl 
-r--r--r--   1 nick     users         1874 Jan 19 10:23 socktest.pl

 

Strange numbers…

 

You might have encountered things like chmod 755 somefile and of course you will be wondering what this is. The thing is, that you can change the entire permission pattern of a file in one go using one number like the one in this example. Every mode has a corresponding code number, and as we shall see there is a very simple way to figure out what number corresponds to any mode.

 

Every one of the three digits on the mode number corresponds to one of the three permission triplets. (u, g and o) Every permission bit in a triplet corresponds to a value: 4 for r, 2 for w, 1 for x. If the permission bit you add this value to the number of the permission triplet. If it is cleared, then you add nothing. (Some of you might notice that in fact, the number for a triplet is the octal value corresponding to the three-bit pattern – if you don’t know what an octal value is, it doesn’t really matter, just follow the intstructions) So if a file has rwxr-xr-x permissions we do the following calculation:

 

Triplet for u: rwx => 4 + 2 + 1 = 7
Triplet for g: r-x => 4 + 0 + 1 = 5
Tripler for o: r-x => 4 + 0 + 1 = 5
Which makes : 755

 

So, 755 is a terse way to say ‘I don’t mind if other people read or run this file, but only I should be able to modify it’ and 777 means‘everyone has full access to this file’

 

 

FROM HERE

My Basic VI commands

vi filename  edit filename starting at line 1
:q  quit (or exit) vi

:q!  quit vi even though latest changes have not been saved for this vi call

:0 or 1G move cursor to first line in file
:n or nG move cursor to line n
:$ or G move cursor to last line in file
i insert text before cursor, until  hit
/string search forward for occurrence of string in text

 n move to next occurrence of search string

N move to next occurrence of search string in opposite direction

:e  refresh the currently loaded file

CTRL-u Moves screen up ½ page
CTRL-d Moves screen down ½ page
CTRL-b Moves screen up one page
CTRL-f Moves screen down one page

Save&Quit

:w    write to disk
     ZZ    write to disk and exit
     :q!   exit without writing to disk

Actually, the command for quitting vi is :q. You can save and quit by typing :wq but ZZ does the same thing1 and takes one less keystroke. If there are unsaved changes to the text and you try to quit using :qvi will warn you that you have unsaved changes and will prevent you from quitting. In order to quit without saving the changes you must use the override switch, !.

For the purist, :wq and ZZ are not exactly the same.  :wq always saves, whereas ZZ saves only if changes have been made since the last save.

To paste from clipboard: First, make sure you’re in edit mode (press i). Then you can paste with Ctrl+Shift+V

remove a char -> x

delete the line -> dd

delete the rest of the line -> D

start and end of the line

start of the line -> 0  OR ^

end of the line  -> $

count occurrence

this would take advantage of the substitute() command.

% is for the entire file

s is for substituion

g is global,

n is for making no change to the document.

:%s/pattern//gn
OR word:
:%s/\<word\>//gn

undo

use

u

cannot save file due to not-root/not-sudo

In the page Top Ten One-Liners from CommandLineFu Explained is suggested this trick (the #3):

:w !sudo tee %

this write the current buffer to the stdin of the command after the !. The % symbol is substituted with the current filename.

show line numbers

:set nu

delete multiple lines

  1. from current line to the end: dG
    Just to note, the action d indicates delete, and G indicates the last line of the file. So, while at any line you press dG, it deletes all the line starting from the current line till the last one.
  2. from current line to certain line, first we show line number using the above command, let say we want to delete from 55 to 100, we first :set nu to see the line number and then navigate to line 55, then we do d100G to remove the line from current to 100.

in the last line and edit

Often time, we use G and in the last line, if I want to start a new line to edit, I have to use $ to the end and press i to edit mode and then move cursor to right and then hit enter to the new line. Turns out i could use one command to do all above:

o Open a new line after current line  // this IS the solution!!
O Open a new line before current line

..

FROM HERE