Quick Language Reference: Bash

general shell debug flags:

    sh -n    check syntax
    sh -v    echos commands before executing
    sh -x    echos commands after running them
    sh -u    gives an error message

define variable

    a=1

to reference a variable, prepend it with $ or wrap with ${}

    echo "your var is $a"
    echo "your var is ${a}"   # protects from ambiguity from connecting chars

    like perl/php, variables in a double quoted string are interperlated.

to make the variable available to sub shells

    export a=1

    # spawn new shell
    bash

    echo $a
    # prints 1

note: executing a script, or a pipe will typically create a new shell
avoid sub shells if you don't need them

prevent string interpolation

    use single quotes

    script 'a$s'


escape single quote

    '"'"'
or
    '\''

    "don't"

    # use double quotes only on string
    'don'"'"'t'

    'don'\''t'


dynamic string arguments

The interpreter is smart enough to pass along a string that's already parsed, to the next command.  you do NOT need quotes around interpolated strings:

    example:

        # does not work ... too many single quotes
        file=.
        args=" -a '$file' "
        ls $args
        # fails.  looking for a file named \'.\'

        # works ... no single quotes within string:
        file=.
        args=" -a $file "
        ls $args

    summary: The only time you need to encapsulate string data in quotes is on the initial shell call (initial string parse). ... I think :)


add numbers

    num=$((num1 + num2))
    num=$(($num1 + $num2))       # also works
    num=$((num1 + 2 + 3))        # ...
    num=$[num1+num2]             # old, deprecated arithmetic expression syntax


error handling / exception handling

    # $? reports last error.
    # WARNING, IT IS CONSUMED ON FIRST TEST, SINCE EACH COMMAND WRITES TO IT

        grep 23412341  .
        grep: .: Is a directory

        $ echo $?
        2

        $ echo $?
        0

    # example test
    if [[ $? == 0 ]]
    then
        java Test
    fi


define / create an alias (bash syntax)

   
    alias name=value
    alias name2='command'
    alias name3='command arg1 arg2'
    alias name4='/path/to/script'
    alias name5='/path/to/script.pl arg1'


disable alias that overrides command, use ''

    'ls'

    unalias aliasname


alternate string quote syntax

c
    a = $'te\'st'

        strings that are scanned for ansi c like escape sequences.

i18n

    a = $"test"

    means get the international text. if there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign simply is ignored, which results in a normal double-quoted string.



copy var

    b=$a
    echo $b


if then, else if, else

    # test if variable is NOT set
    if [ -z "$VAR" ]; then

    # else if
    elif [ -z "$VAR2" ]; then

    else
   
   
    # end
    fi

    # note you can also write the 'then' on its own line


negation

    !     means not


test variable has a value

    if [ -z "$VAR" ];
    then
        echo "value is not set"
    fi


test file existence:


    if [ -f $FILE ];
    then
       echo "File $FILE exists."
    else
       echo "File $FILE does not exist."
    fi

    if [ ! -f $FILE ];
    then
       echo "File $FILE does not exist."
    fi


test if file contains pattern

    if grep -q FOOBAR "$File"; then
       # file has FOOBAR
    fi

test if file does not contain pattern

    if ! grep -q SomeString "$File"; then
       # Some Actions
    fi


then: can be written on the same line with a ; or on the next line

    # people likely do this because veritical space is limited on an 80x24 terminal
    if ! grep -q SomeString "$File"; then
        echo 1;
    fi

    # means the same as
    if ! grep -q SomeString "$File"
    then
        echo 1;

    fi


shell redirection

    cmd 2>      redirect stderr
    cmd &>      redirect both stderr and stdout

    cmd >&n     send output to descriptor n
    cmd m>&n    redirect output that would have gone to m to n
    cmd <&n     get input from descriptor n
    cmd <&-     close standard in


silence output


    # SILENCIO POR FAVOR!
    cmd &> /dev/null


pipes | are used to chain commands together.  output of the first command is wired up to the input of the next command (like snapping together legos)

    comand | tee file.txt   prints to screen and saves to file.txt


edit previous command

    ^foo^bar        executes previous command, except replaces foo with bar


loops:

    for loop

        for ((i=1;i<=100;i++));
        do
           # your-unix-command-here
           echo $i
        done
   
    while loop

        while ((1));
        do
           # your-unix-command-here
            sleep 1;
        done
       

    for

         for x in file1 file2 file3
         do
         sed 's/thier/their/g' $x > ,$x
         mv ,$x $x
         done
       

    # index iteration
     for ((i=0; i<10; i++)); do
         echo $i
     done


bash heredoc syntax

    # note no ; required
    # for quoting multiline strings
    cat << ENDOFBLOCK
            sdfg
ENDOFBLOCK

    Note: you can't assign this straight to a var
    For example, you can use a cat do

    variable=$(cat <<EOF
<html>
<body>
<p>THis is a test</p>
</body>
</html>
EOF
)


test loop

    You can preview what loops will do by putting an echo in front of the commands.  you don't want to realize there was a mistake after running the command a thousand times.

        # test
        for a in *
        do
            echo mv a b
        done

        # then use the command
        mv a b
   

    This can mess up " in output though, since echo will interpret it and discard.   if you need exact output, wrap the block in a literal heredoc.  THis will preserve all quotes " ' ` and meta chars

    So, intest you can use a heredoc to prevent interpolation

    cat <<DEBUG
    echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `
DEBUG
    # prints:
    #     echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `


piping output from one command to a loop

    find photos/ | while read file; do echo $file ; done;


loops in bash and variable scope:

  counter=0
  for f in $(find .)
  do
    let counter+=1
  done
  echo $counter

problem: if ls returns a large result, can overflow the buffer.

      counter=0
      find . | while read f
      do
        let counter+=1
      done
      echo $counter


    the problem: variables inside loop in their own shell, can't be used if needed.

    For read only access, use export in the outer context

        export var=1

    Though, this won't work if you need write access.  In this case you need < <() syntax
    for example


      counter=0
      while read f
      do
        let counter+=1
      done < <(find .)
      echo $counter

    ugly syntax, but works


handle whitespace in filename

    find <path> -mtime +7 | while read line
    do
        # just wrap with double quotes
        mv "$line" /somewhere/else
    done

you can embed a comment string in the heredoc marker

    # comment following line out after testing
    cat <<'#ENDDEBUG'

    echo PARAM=`ls -la  | awk '{print $3}' `
    #ENDDEBUG



parse delimted test with `cut`

    # list all users defined on system
    # split on :, print the first field
    cut -d: -f1 /etc/passwd


increment var

  y=$[y+1]
  y=$(($y+1))
  ((y++))


polling process (repeatedly testing)

    while (( 1 ));
     do   ls -lah sedOSEZA8 ;
     sleep 1;
    done


see all characters used in a file

    # change char to char\n
    cat Configuration | sed -e 's/\(.\)/\1\n/g' | sort | uniq



create a simple index page for .html files in a folder

 ls *.html -1 | while read f ; do echo "<a href='$f'>$f</a><br>"; done >> index.htm


scp - secure copy (use instead of ftp)

  scp  d_HomeBk.sql  user@server:/remote/path/to/dir/

  note: if there are spaces in file name use: encased/escaped quotes "\"  \""

    eg  

      scp  d_HomeBk.sql  user@server:"\"/path/to/file/\""

    or enclose entire string in single quotes, with double

      scp  d_HomeBk.sql  'user@server:"/path/to/file/"'

    use -r to copy recursively



find files

    # last 24 hours
    find . -mtime 0

    # older than a month
    find . -mtime +31

    # find files only
    find -type f


when backticks overflow command line, can group in blocks and pipe to a loop:

    % sh
    find . -type f -mtime -1 -print |
    fmt -1000 |
    while read files
    do pr -n $files
    done | lpr
    exit

    ...or use xargs
    xargs lpr < allfiles.tmp

or

    # NOTE the -I defines a string to replace
    # default behavior sends to first argument of command.
    command | xargs -I '{}' somecommand stuff '{}'
    command | xargs rm -rf


# copy a lot of image files, convert sizes


    ls -1 | while read file; do echo $file; \
    convert -resize 460 "$file" "/cygdrive/c/temp/yourfiles/$file" ; done


# slightly better complex ... can be stopped, restarted

    $destdir = /cygdrive/c/temp/yourfiles

    ls -1 | while read $file
    do
      if [[ ! -e $destdir/$file ]]
      then
        echo $file
        convert -resize 460 $file $destdir/$file
      fi
    done



internal file separator.  note: also there is the internal variable IFS (internal file seperator) that is noramally set to white space

    while can parse a line, such as

    IFS=':'
    cat  /etc/passwd | while read var1 var2 var3 var4
    do
     ...
    done

    # although awk is probably the better tool for this



the previous file edited on bash shell

    !$    





convert stream to lower case:

    alias lc="tr '[:upper:]' '[:lower:]'"
    alias uc="tr '[:lower:]' '[:upper:]'"
 
    eg:

     cat file.txt | lc



copy all but some files

    cpio can copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .snapshot (and anything in them).

    cd /source-dir
    find . -name .snapshot -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


    This command copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .svn (and anything in them).
    cd /source-dir
    find . -name .svn -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


profile vs bashrc

    profiles are for interactive login shells (that requre user/pass),
    rc is for non-interactive, non-login


mount cd rom manullly

     mkdir -p /mnt/cdrom && mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom


functions

                # without arg, called like a command line tool
                f() { echo 1; };
                f;


                # with argument
                f() { echo $1; };
                f arg ;


floating point numbers

    bwa hahahah ... :-)
    bash doesn't support floating point arithmetic evaluation, but you can use the pretty-standard bc command in your function to help out.     example:

    function f_to_c { echo "5/9 * ($1 - 32)" | bc -l; }

    Though if your script is getting complex enough to use floating point numbers, it probably should be written in an actual scripting language (python, perl, ruby, php, etc) not bash.

arrays:

    If you are using arrays, you should start to wonder if you should be using bash at all and not another programming language.  :)
    But yes, you can do arrays.

    #simple array:

        name[index]=value

    # explicitly define numeric array


        declare -a chars  
        chars[0]="a"
        chars[1]="b"

       echo ${chars[0]}


    # set list of values

         declare -a arrayname=(element1 element2 element3)

    # bash shorthand: declare/assign array

        arr=(Hello World)

    # in referencing the value, you must enclose in {}

          echo ${arr[0]} ${arr[1]}

    To quote from the man page:
    The braces are required to avoid conflicts with pathname expansion.
    In addition the following funky constructs are available:

          ${arr[*]}         # All of the items in the array
          ${!arr[*]}        # All of the indexes in the array
          ${#arr[*]}        # Number of items in the array
          ${#arr[0]}        # Length of item zero

    # associative array or hash

        declare -A morse  
        morse[a]="dot; dash"
        morse[b]="dash; dot; dot; dot"
        morse[c]="dash; dot; dash; dot"
        # ...

       echo ${morse[$1]}   # write dots, dashes for letter

    # print all of array, use @ (index number) or * (all)

        echo ${morse[@]}

    # to also export declared vars

        declare -x ....


eval "string"

    execute code.  similar to `` or $(), but may read a bit easier

case:  (switch statement)

    Note: no break/exit needed

    # number of feet
    case $ANIMAL in

      horse | dog | cat) echo -n "four";;

      man | kangaroo ) echo -n "two";;

      alien ) echo -n "pi";;

      *)
        #default
        echo -n "an unknown number";;

    esac

print current user

    echo "$USER"
    id -u -n

    To print numeric UID, run:

    id -u

change passwd non-interactive

   
    # reads a list of users passwords (non interactive)
    # may be encrypted
    echo "username:newpass" | chpasswd


find all files not owned by user

    find '!' -uid 0
    find '!' -user root


can also  echo multiline strings

    echo "
    /var/log/top.log {
        missingok
    }
    " > /etc/logrotate.d/top
   

to avoid pipe in main loop

    this:

        while read log
        do
          echo $log
        done < <( svn log $1 | grep -E -e "^r[0-9]+ \| " | tac )

    is mostly the same as:

        svn log $1 | grep -E -e "^r[0-9]+ \| " | tac | while read log
        do
          echo $log
        done

ssh

    Warning: ssh reads from standard input, and can "eat" all your input if used in a while loop.
    Instead, in a loop use:

        cat /dev/null | ssh $u@$h  $COMMAND

    or
         ssh -n


    In general, you should close or redirect stdin for the other commands you
    run, to stop them reading from the file.

        sh call_other_script.sh </dev/null


rotate file based on size

    #!/bin/bash

    f=/var/log/top.log
    date         >> $f
    echo 'top:'  >> $f
    top -b -n 1  >>  $f

    echo '--'    >> $f
    echo 'ps:'   >> $f

    ps auxw      >>  $f

    echo '--'    >> $f
    echo ''      >> $f


    # don't let it get too big
    FILESIZE=$(stat -c%s "$f")
    MAXSIZE=10000000

    if [ $FILESIZE -gt $MAXSIZE ]; then
      mv -f $f $f.1
    fi



count word frequency

    tr ' ' '\n' < /tmp/kevtest_old.txt | sort | uniq -c | sort -nr | head

    27306
    5200 69
    1980 137
    1716 129
     728 93
     611 56
     611 55
     520 42
     520 41
     248 70

    alt:

        awk '{a[$1]++}END{for(k in a)print a[k],k}' RS=" |\n" file > myfile



test string for pattern


    build=123000
    if [[ $build =~ "3" ]]
    then
        echo "has 3"
    fi

    build=123000
    if [[ $build =~ "4" ]]
    then
        echo "has 4"
    fi


convert string list to array

    STRING="test1 test2 test3"

    # convert to array
    list=( $STRING )

    #Or more verbosely:

    declare -a list=( $STRING )

    PS: You can't export IFS and use the new value in the same command. You have to declare it first, then use it's effects in the following command:

    list=( first second third )
    echo "${list[*]}"


    #first element
    list=( first second third )
    echo "${list[0]}"
    # first


nohup


    # don't stop command if the terminal connction is lost

        nohup  one_command_only  &

    #execute multiple commands, call script in future

        # get both process ids
        function sleep_restart() {
                # to stagger node restarts, call script in future
                sleep=1
                echo "Will schedule start in $sleep seconds..."
                echo ""
                echo "    sleep $sleep && /yourcommand_restart"
                echo ""
                nohup sh -c "sleep $sleep && /yourcommand_restart >& /dev/null" &
                sleep 1
        }


find by permission

    find . -perm /222

    Search for files which are writable by somebody (their owner, or their group, or anybody else).

    find . -perm /220
    find . -perm /u+w,g+w
    find . -perm /u=w,g=w

    find . -perm /o=w


find loops in symlinks

    find . -follow -printf ""

find number of processes

    nproc
    cat /proc/cpuinfo

# count words or count repeated lines by occurance

    sort | uniq -c



remove XML tags, remove HTML tags

    sed -n '/^$/!{s/<[^>]*>//g;p;}'


base64 decode

    base64 --decode

    # eg, convert xml 64 data to jpeg
    curl  <YOUR_URL_HERE>  | sed -n '/^$/!{s/<[^>]*>//g;p;}' | base64 --decode > test.jpg


simple tail log, execute trigger when it reads a pattern

    # look for lines containing OutOfMemoryError
    log=/tmp/kevtest.txt
    pattern="OutOfMemoryError"
    while read line;
    do
        if [[ $line =~ $pattern  ]]
        then
            echo "hit $line !"
            break;
        fi
    done < <(tail -f $log)


    alt: with awk

    # no exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1");  }'

    # with exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1"); exit; }'


hex encoding to ascii

    d=`cat lib.php `

    echo -e $d  | awk '{printf "%s\n", $_}' > kevtest.php


test if socket port is open

    Use netcat (nc) for the client and server tests

    # basic interactive chat

        # server (use same ports on client and server)
        nc -l 7601 -v

        # client (use same ports on client and server)
        nc <HOST> 7601

        # note: exit with ctrl-C


    # scan port (non-interactive)

        nc -z <host> <port>


    # --
    # client examples
    # try socket connection
    # interactive

        nc  <host> <port>

    or (older)

        telnet  <host> <port>

    #scan (one shot, not interactive)

        nc -z <host> <port>

    # For a quick non-interative check (with a 5 seconds timeout):

        nc -z -w5 <host> <port>


    # --
    # SERVER
    # note: low number ports (such as 80) require higher perms
    # create socket that stays open indefinitely until manually closed
    #listen locally on port 1024 for all traffic and dump the output to STDOUT

        nc -l 1024 -k

    # more verbose

        nc -l 1024 -k  -v

    # close after first message

        nc -l 1024


    # more complex request (send mail)

        nc localhost 25 << EOF
        HELO host.example.com
        MAIL FROM: <user@host.example.com>
        RCPT TO: <user2@host.example.com>
        DATA
        Body of email.
        .
        QUIT
        EOF


--
BASH BRACKETS/BRACES/PARANS

There are a lot of brackets in Bash, like () (()) [] [[]], and some have overloaded meanings, which can get confusing.  Here's a quick overview of what they mean.  These types of symbols are hard to search for explanations as well.


bracket options in 'if' statements

    no brackets        execute shell commands and tests exit code
                    0 == true in this case, which can be counter-intuitive
                    'exit code' should really be named 'error code', so
                    'no errors' means true.  last code is stored in variable $?

    [ ] are simpler, older, POSIX compliant.
         it is usually just an alias to /bin/test, /bin/[

    [[ ]]  the double brackets are builting bash keywords

        in general use [[  vs [ in tests if you don't need to port to other shells

        it's newer and processes internally (better for whitespace handling)

        The double bracket enables additional functionality.
        For example, you can use && and || instead of -a and -o
        and there's a regular expression matching operator =~

    (( )) are used for arithemetic operations.  for example
            a=((a++))
            for ((i=0; i<10; i++)); do echo $i; done;


    $[ ]     deprecated syntax meaning (( ))


other bracket contexts


    ( )        1. are used to create a subshell, won't affect current shell
            ( cd /tmp; pwd ); pwd;

            2. are also used to create arrays, [ ] used for array index
                 array=(1 2 3)
                 echo ${array[1]}

            3. are also used used to define functions.

                # without arg, called like a command line tool
                f() { echo 1; }; f;
                # with argument
                f() { echo $1; }; f arg ;


            4. process substitution is:

                  cmd <(list)   is similar to        cat list | cmd
                or
                  cmd > >(list)    is similar to        cmd | list


                 The effect of process substitution is to make each list act like a file.

                however >() is more powerful than a normal pipe:
                You can't use | to redirect standard output and standard error to different programs.


    { }    does a couple things:
            1. unambiguously identifies variables. example ${a}${b}
            2. executes a sequence of commands in the CURRENT shell context, opposite of ()

                { date; top -b -n1 | head ; } >logfile

            There is a subtle syntactic difference with ( ), though (see bash reference) ; essentially, a semicolon ; after the last command within braces is a must,
            and the braces {, } must be surrounded by spaces.

    [ ] 1. can mean a range operator, or matching list

            ls [abc][1-4]
           
        2. can be the index designator in an array  ${a[0]}
               

    $( )     runs a command and drops the output in another.  like backticks ``
            usefule for nesting operations, example:

                # find string 'bar' in files named 'foo'
                grep bar $( find -name foo )




--
BASH STRING MANIPULATION

strip extension

    nameis=${filename%.php}
    nameis=${filename%.*}  # general


substring examples:

    The variable var contains /a/b/c/d/e.f.g:


        Expression   Result

        ${var}   /a/b/c/d/e.f.g

        # match first from left
        ${var#/*/}   b/c/d/e.f.g

        #match last from left
        ${var##/*/}   e.f.g

        # match first from right  
        ${var%.*}   /a/b/c/d/e.f

        # match last from right
        ${var%%.*}   /a/b/c/d/e

        # misc
        ${var%%/*/}   /a/b/c/d/e.f.g
        ${var%%/*}
        ${var%/b*}   /a
        ${var%%/b*}   /a


by length

    stringZ=abcABC123ABCabc
    #       0123456789.....
    #       0-based indexing.

    echo ${stringZ:0}                            # abcABC123ABCabc
    echo ${stringZ:1}                            # bcABC123ABCabc
    echo ${stringZ:7}                            # 23ABCabc

    echo ${stringZ:7:3}                          # 23A
                                                 # Three characters of substring.



    # Is it possible to index from the right end of the string?
       
    echo ${stringZ:-4}                           # abcABC123ABCabc
    # Defaults to full string, as in ${parameter:-default}.
    # However . . .

    echo ${stringZ:(-4)}                         # Cabc
    echo ${stringZ: -4}                          # Cabc
    # Now, it works.
    # Parentheses or added space "escape" the position parameter.



find replace (substring replacement)

    Replace first match of $substring with $replacement:

        ${string/substring/replacement}

    Replace all matches of $substring with $replacement.

        ${string//substring/replacement}




bash string editing operators

    Operator        Explanation

    ${variable#pattern}    Delete the shortest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable##pattern}    Delete the longest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable%pattern}    Delete the shortest part of pattern that matches the end of variable's value.Return the rest.
    ${variable%%pattern}    Delete the longest part of pattern that matches the end of variable's value.Return the rest.

    The patterns can be filename wildcard characters: *, ?, and []; with string editing operators, wildcards match strings in the same way they match filenames. (These are not sed-like regular expressions.) The first two operators, with #, edit variables from the front. The other two, with %, edit from the end. Here's a system for remembering which does what: you put a number sign (#) at the front of a number and a percent sign (%) at the end of a number.