Sunday, November 22, 2015

Quick Language Reference: Bash

general shell debug flags:

    sh -n    check syntax
    sh -v    echos commands before executing
    sh -x    echos commands after running them
    sh -u    gives an error message

define variable


to reference a variable, prepend it with $ or wrap with ${}

    echo "your var is $a"
    echo "your var is ${a}"   # protects from ambiguity from connecting chars

    like perl/php, variables in a double quoted string are interperlated.

to make the variable available to sub shells

    export a=1

    # spawn new shell

    echo $a
    # prints 1

note: executing a script, or a pipe will typically create a new shell
avoid sub shells if you don't need them

prevent string interpolation

    use single quotes

    script 'a$s'

escape single quote



    # use double quotes only on string


dynamic string arguments

The interpreter is smart enough to pass along a string that's already parsed, to the next command.  you do NOT need quotes around interpolated strings:


        # does not work ... too many single quotes
        args=" -a '$file' "
        ls $args
        # fails.  looking for a file named \'.\'

        # works ... no single quotes within string:
        args=" -a $file "
        ls $args

    summary: The only time you need to encapsulate string data in quotes is on the initial shell call (initial string parse). ... I think :)

add numbers

    num=$((num1 + num2))
    num=$(($num1 + $num2))       # also works
    num=$((num1 + 2 + 3))        # ...
    num=$[num1+num2]             # old, deprecated arithmetic expression syntax

error handling / exception handling

    # $? reports last error.

        grep 23412341  .
        grep: .: Is a directory

        $ echo $?

        $ echo $?

    # example test
    if [[ $? == 0 ]]
        java Test

define / create an alias (bash syntax)

    alias name=value
    alias name2='command'
    alias name3='command arg1 arg2'
    alias name4='/path/to/script'
    alias name5='/path/to/ arg1'

disable alias that overrides command, use ''


    unalias aliasname

alternate string quote syntax

    a = $'te\'st'

        strings that are scanned for ansi c like escape sequences.


    a = $"test"

    means get the international text. if there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign simply is ignored, which results in a normal double-quoted string.

copy var

    echo $b

if then, else if, else

    # test if variable is NOT set
    if [ -z "$VAR" ]; then

    # else if
    elif [ -z "$VAR2" ]; then

    # end

    # note you can also write the 'then' on its own line


    !     means not

test variable has a value

    if [ -z "$VAR" ];
        echo "value is not set"

test file existence:

    if [ -f $FILE ];
       echo "File $FILE exists."
       echo "File $FILE does not exist."

    if [ ! -f $FILE ];
       echo "File $FILE does not exist."

test if file contains pattern

    if grep -q FOOBAR "$File"; then
       # file has FOOBAR

test if file does not contain pattern

    if ! grep -q SomeString "$File"; then
       # Some Actions

then: can be written on the same line with a ; or on the next line

    # people likely do this because veritical space is limited on an 80x24 terminal
    if ! grep -q SomeString "$File"; then
        echo 1;

    # means the same as
    if ! grep -q SomeString "$File"
        echo 1;


shell redirection

    cmd 2>      redirect stderr
    cmd &>      redirect both stderr and stdout

    cmd >&n     send output to descriptor n
    cmd m>&n    redirect output that would have gone to m to n
    cmd <&n     get input from descriptor n
    cmd <&-     close standard in

silence output

    cmd &> /dev/null

pipes | are used to chain commands together.  output of the first command is wired up to the input of the next command (like snapping together legos)

    comand | tee file.txt   prints to screen and saves to file.txt

edit previous command

    ^foo^bar        executes previous command, except replaces foo with bar


    for loop

        for ((i=1;i<=100;i++));
           # your-unix-command-here
           echo $i
    while loop

        while ((1));
           # your-unix-command-here
            sleep 1;


         for x in file1 file2 file3
         sed 's/thier/their/g' $x > ,$x
         mv ,$x $x

    # index iteration
     for ((i=0; i<10; i++)); do
         echo $i

bash heredoc syntax

    # note no ; required
    # for quoting multiline strings
    cat << ENDOFBLOCK

    Note: you can't assign this straight to a var
    For example, you can use a cat do

    variable=$(cat <<EOF
<p>THis is a test</p>

test loop

    You can preview what loops will do by putting an echo in front of the commands.  you don't want to realize there was a mistake after running the command a thousand times.

        # test
        for a in *
            echo mv a b

        # then use the command
        mv a b

    This can mess up " in output though, since echo will interpret it and discard.   if you need exact output, wrap the block in a literal heredoc.  THis will preserve all quotes " ' ` and meta chars

    So, intest you can use a heredoc to prevent interpolation

    cat <<DEBUG
    echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `
    # prints:
    #     echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `

piping output from one command to a loop

    find photos/ | while read file; do echo $file ; done;

loops in bash and variable scope:

  for f in $(find .)
    let counter+=1
  echo $counter

problem: if ls returns a large result, can overflow the buffer.

      find . | while read f
        let counter+=1
      echo $counter

    the problem: variables inside loop in their own shell, can't be used if needed.

    For read only access, use export in the outer context

        export var=1

    Though, this won't work if you need write access.  In this case you need < <() syntax
    for example

      while read f
        let counter+=1
      done < <(find .)
      echo $counter

    ugly syntax, but works

handle whitespace in filename

    find <path> -mtime +7 | while read line
        # just wrap with double quotes
        mv "$line" /somewhere/else

you can embed a comment string in the heredoc marker

    # comment following line out after testing
    cat <<'#ENDDEBUG'

    echo PARAM=`ls -la  | awk '{print $3}' `

parse delimted test with `cut`

    # list all users defined on system
    # split on :, print the first field
    cut -d: -f1 /etc/passwd

increment var


polling process (repeatedly testing)

    while (( 1 ));
     do   ls -lah sedOSEZA8 ;
     sleep 1;

see all characters used in a file

    # change char to char\n
    cat Configuration | sed -e 's/\(.\)/\1\n/g' | sort | uniq

create a simple index page for .html files in a folder

 ls *.html -1 | while read f ; do echo "<a href='$f'>$f</a><br>"; done >> index.htm

scp - secure copy (use instead of ftp)

  scp  d_HomeBk.sql  user@server:/remote/path/to/dir/

  note: if there are spaces in file name use: encased/escaped quotes "\"  \""


      scp  d_HomeBk.sql  user@server:"\"/path/to/file/\""

    or enclose entire string in single quotes, with double

      scp  d_HomeBk.sql  'user@server:"/path/to/file/"'

    use -r to copy recursively

find files

    # last 24 hours
    find . -mtime 0

    # older than a month
    find . -mtime +31

    # find files only
    find -type f

when backticks overflow command line, can group in blocks and pipe to a loop:

    % sh
    find . -type f -mtime -1 -print |
    fmt -1000 |
    while read files
    do pr -n $files
    done | lpr

    ...or use xargs
    xargs lpr < allfiles.tmp


    # NOTE the -I defines a string to replace
    # default behavior sends to first argument of command.
    command | xargs -I '{}' somecommand stuff '{}'
    command | xargs rm -rf

# copy a lot of image files, convert sizes

    ls -1 | while read file; do echo $file; \
    convert -resize 460 "$file" "/cygdrive/c/temp/yourfiles/$file" ; done

# slightly better complex ... can be stopped, restarted

    $destdir = /cygdrive/c/temp/yourfiles

    ls -1 | while read $file
      if [[ ! -e $destdir/$file ]]
        echo $file
        convert -resize 460 $file $destdir/$file

internal file separator.  note: also there is the internal variable IFS (internal file seperator) that is noramally set to white space

    while can parse a line, such as

    cat  /etc/passwd | while read var1 var2 var3 var4

    # although awk is probably the better tool for this

the previous file edited on bash shell


convert stream to lower case:

    alias lc="tr '[:upper:]' '[:lower:]'"
    alias uc="tr '[:lower:]' '[:upper:]'"

     cat file.txt | lc

copy all but some files

    cpio can copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .snapshot (and anything in them).

    cd /source-dir
    find . -name .snapshot -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir

    This command copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .svn (and anything in them).
    cd /source-dir
    find . -name .svn -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir

profile vs bashrc

    profiles are for interactive login shells (that requre user/pass),
    rc is for non-interactive, non-login

mount cd rom manullly

     mkdir -p /mnt/cdrom && mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom


                # without arg, called like a command line tool
                f() { echo 1; };

                # with argument
                f() { echo $1; };
                f arg ;

floating point numbers

    bwa hahahah ... :-)
    bash doesn't support floating point arithmetic evaluation, but you can use the pretty-standard bc command in your function to help out.     example:

    function f_to_c { echo "5/9 * ($1 - 32)" | bc -l; }

    Though if your script is getting complex enough to use floating point numbers, it probably should be written in an actual scripting language (python, perl, ruby, php, etc) not bash.


    If you are using arrays, you should start to wonder if you should be using bash at all and not another programming language.  :)
    But yes, you can do arrays.

    #simple array:


    # explicitly define numeric array

        declare -a chars  

       echo ${chars[0]}

    # set list of values

         declare -a arrayname=(element1 element2 element3)

    # bash shorthand: declare/assign array

        arr=(Hello World)

    # in referencing the value, you must enclose in {}

          echo ${arr[0]} ${arr[1]}

    To quote from the man page:
    The braces are required to avoid conflicts with pathname expansion.
    In addition the following funky constructs are available:

          ${arr[*]}         # All of the items in the array
          ${!arr[*]}        # All of the indexes in the array
          ${#arr[*]}        # Number of items in the array
          ${#arr[0]}        # Length of item zero

    # associative array or hash

        declare -A morse  
        morse[a]="dot; dash"
        morse[b]="dash; dot; dot; dot"
        morse[c]="dash; dot; dash; dot"
        # ...

       echo ${morse[$1]}   # write dots, dashes for letter

    # print all of array, use @ (index number) or * (all)

        echo ${morse[@]}

    # to also export declared vars

        declare -x ....

eval "string"

    execute code.  similar to `` or $(), but may read a bit easier

case:  (switch statement)

    Note: no break/exit needed

    # number of feet
    case $ANIMAL in

      horse | dog | cat) echo -n "four";;

      man | kangaroo ) echo -n "two";;

      alien ) echo -n "pi";;

        echo -n "an unknown number";;


print current user

    echo "$USER"
    id -u -n

    To print numeric UID, run:

    id -u

change passwd non-interactive

    # reads a list of users passwords (non interactive)
    # may be encrypted
    echo "username:newpass" | chpasswd

find all files not owned by user

    find '!' -uid 0
    find '!' -user root

can also  echo multiline strings

    echo "
    /var/log/top.log {
    " > /etc/logrotate.d/top

to avoid pipe in main loop


        while read log
          echo $log
        done < <( svn log $1 | grep -E -e "^r[0-9]+ \| " | tac )

    is mostly the same as:

        svn log $1 | grep -E -e "^r[0-9]+ \| " | tac | while read log
          echo $log


    Warning: ssh reads from standard input, and can "eat" all your input if used in a while loop.
    Instead, in a loop use:

        cat /dev/null | ssh $u@$h  $COMMAND

         ssh -n

    In general, you should close or redirect stdin for the other commands you
    run, to stop them reading from the file.

        sh </dev/null

rotate file based on size


    date         >> $f
    echo 'top:'  >> $f
    top -b -n 1  >>  $f

    echo '--'    >> $f
    echo 'ps:'   >> $f

    ps auxw      >>  $f

    echo '--'    >> $f
    echo ''      >> $f

    # don't let it get too big
    FILESIZE=$(stat -c%s "$f")

    if [ $FILESIZE -gt $MAXSIZE ]; then
      mv -f $f $f.1

count word frequency

    tr ' ' '\n' < /tmp/kevtest_old.txt | sort | uniq -c | sort -nr | head

    5200 69
    1980 137
    1716 129
     728 93
     611 56
     611 55
     520 42
     520 41
     248 70


        awk '{a[$1]++}END{for(k in a)print a[k],k}' RS=" |\n" file > myfile

test string for pattern

    if [[ $build =~ "3" ]]
        echo "has 3"

    if [[ $build =~ "4" ]]
        echo "has 4"

convert string list to array

    STRING="test1 test2 test3"

    # convert to array
    list=( $STRING )

    #Or more verbosely:

    declare -a list=( $STRING )

    PS: You can't export IFS and use the new value in the same command. You have to declare it first, then use it's effects in the following command:

    list=( first second third )
    echo "${list[*]}"

    #first element
    list=( first second third )
    echo "${list[0]}"
    # first


    # don't stop command if the terminal connction is lost

        nohup  one_command_only  &

    #execute multiple commands, call script in future

        # get both process ids
        function sleep_restart() {
                # to stagger node restarts, call script in future
                echo "Will schedule start in $sleep seconds..."
                echo ""
                echo "    sleep $sleep && /yourcommand_restart"
                echo ""
                nohup sh -c "sleep $sleep && /yourcommand_restart >& /dev/null" &
                sleep 1

find by permission

    find . -perm /222

    Search for files which are writable by somebody (their owner, or their group, or anybody else).

    find . -perm /220
    find . -perm /u+w,g+w
    find . -perm /u=w,g=w

    find . -perm /o=w

find loops in symlinks

    find . -follow -printf ""

find number of processes

    cat /proc/cpuinfo

# count words or count repeated lines by occurance

    sort | uniq -c

remove XML tags, remove HTML tags

    sed -n '/^$/!{s/<[^>]*>//g;p;}'

base64 decode

    base64 --decode

    # eg, convert xml 64 data to jpeg
    curl  <YOUR_URL_HERE>  | sed -n '/^$/!{s/<[^>]*>//g;p;}' | base64 --decode > test.jpg

simple tail log, execute trigger when it reads a pattern

    # look for lines containing OutOfMemoryError
    while read line;
        if [[ $line =~ $pattern  ]]
            echo "hit $line !"
    done < <(tail -f $log)

    alt: with awk

    # no exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1");  }'

    # with exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1"); exit; }'

hex encoding to ascii

    d=`cat lib.php `

    echo -e $d  | awk '{printf "%s\n", $_}' > kevtest.php

test if socket port is open

    Use netcat (nc) for the client and server tests

    # basic interactive chat

        # server (use same ports on client and server)
        nc -l 7601 -v

        # client (use same ports on client and server)
        nc <HOST> 7601

        # note: exit with ctrl-C

    # scan port (non-interactive)

        nc -z <host> <port>

    # --
    # client examples
    # try socket connection
    # interactive

        nc  <host> <port>

    or (older)

        telnet  <host> <port>

    #scan (one shot, not interactive)

        nc -z <host> <port>

    # For a quick non-interative check (with a 5 seconds timeout):

        nc -z -w5 <host> <port>

    # --
    # SERVER
    # note: low number ports (such as 80) require higher perms
    # create socket that stays open indefinitely until manually closed
    #listen locally on port 1024 for all traffic and dump the output to STDOUT

        nc -l 1024 -k

    # more verbose

        nc -l 1024 -k  -v

    # close after first message

        nc -l 1024

    # more complex request (send mail)

        nc localhost 25 << EOF
        MAIL FROM: <>
        RCPT TO: <>
        Body of email.


There are a lot of brackets in Bash, like () (()) [] [[]], and some have overloaded meanings, which can get confusing.  Here's a quick overview of what they mean.  These types of symbols are hard to search for explanations as well.

bracket options in 'if' statements

    no brackets        execute shell commands and tests exit code
                    0 == true in this case, which can be counter-intuitive
                    'exit code' should really be named 'error code', so
                    'no errors' means true.  last code is stored in variable $?

    [ ] are simpler, older, POSIX compliant.
         it is usually just an alias to /bin/test, /bin/[

    [[ ]]  the double brackets are builting bash keywords

        in general use [[  vs [ in tests if you don't need to port to other shells

        it's newer and processes internally (better for whitespace handling)

        The double bracket enables additional functionality.
        For example, you can use && and || instead of -a and -o
        and there's a regular expression matching operator =~

    (( )) are used for arithemetic operations.  for example
            for ((i=0; i<10; i++)); do echo $i; done;

    $[ ]     deprecated syntax meaning (( ))

other bracket contexts

    ( )        1. are used to create a subshell, won't affect current shell
            ( cd /tmp; pwd ); pwd;

            2. are also used to create arrays, [ ] used for array index
                 array=(1 2 3)
                 echo ${array[1]}

            3. are also used used to define functions.

                # without arg, called like a command line tool
                f() { echo 1; }; f;
                # with argument
                f() { echo $1; }; f arg ;

            4. process substitution is:

                  cmd <(list)   is similar to        cat list | cmd
                  cmd > >(list)    is similar to        cmd | list

                 The effect of process substitution is to make each list act like a file.

                however >() is more powerful than a normal pipe:
                You can't use | to redirect standard output and standard error to different programs.

    { }    does a couple things:
            1. unambiguously identifies variables. example ${a}${b}
            2. executes a sequence of commands in the CURRENT shell context, opposite of ()

                { date; top -b -n1 | head ; } >logfile

            There is a subtle syntactic difference with ( ), though (see bash reference) ; essentially, a semicolon ; after the last command within braces is a must,
            and the braces {, } must be surrounded by spaces.

    [ ] 1. can mean a range operator, or matching list

            ls [abc][1-4]
        2. can be the index designator in an array  ${a[0]}

    $( )     runs a command and drops the output in another.  like backticks ``
            usefule for nesting operations, example:

                # find string 'bar' in files named 'foo'
                grep bar $( find -name foo )


strip extension

    nameis=${filename%.*}  # general

substring examples:

    The variable var contains /a/b/c/d/e.f.g:

        Expression   Result

        ${var}   /a/b/c/d/e.f.g

        # match first from left
        ${var#/*/}   b/c/d/e.f.g

        #match last from left
        ${var##/*/}   e.f.g

        # match first from right  
        ${var%.*}   /a/b/c/d/e.f

        # match last from right
        ${var%%.*}   /a/b/c/d/e

        # misc
        ${var%%/*/}   /a/b/c/d/e.f.g
        ${var%/b*}   /a
        ${var%%/b*}   /a

by length

    #       0123456789.....
    #       0-based indexing.

    echo ${stringZ:0}                            # abcABC123ABCabc
    echo ${stringZ:1}                            # bcABC123ABCabc
    echo ${stringZ:7}                            # 23ABCabc

    echo ${stringZ:7:3}                          # 23A
                                                 # Three characters of substring.

    # Is it possible to index from the right end of the string?
    echo ${stringZ:-4}                           # abcABC123ABCabc
    # Defaults to full string, as in ${parameter:-default}.
    # However . . .

    echo ${stringZ:(-4)}                         # Cabc
    echo ${stringZ: -4}                          # Cabc
    # Now, it works.
    # Parentheses or added space "escape" the position parameter.

find replace (substring replacement)

    Replace first match of $substring with $replacement:


    Replace all matches of $substring with $replacement.


bash string editing operators

    Operator        Explanation

    ${variable#pattern}    Delete the shortest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable##pattern}    Delete the longest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable%pattern}    Delete the shortest part of pattern that matches the end of variable's value.Return the rest.
    ${variable%%pattern}    Delete the longest part of pattern that matches the end of variable's value.Return the rest.

    The patterns can be filename wildcard characters: *, ?, and []; with string editing operators, wildcards match strings in the same way they match filenames. (These are not sed-like regular expressions.) The first two operators, with #, edit variables from the front. The other two, with %, edit from the end. Here's a system for remembering which does what: you put a number sign (#) at the front of a number and a percent sign (%) at the end of a number.

No comments: