Quick Language Reference: Bash

general shell debug flags:

    sh -n    check syntax
    sh -v    echos commands before executing
    sh -x    echos commands after running them
    sh -u    gives an error message

define variable

    a=1

to reference a variable, prepend it with $ or wrap with ${}

    echo "your var is $a"
    echo "your var is ${a}"   # protects from ambiguity from connecting chars

    like perl/php, variables in a double quoted string are interperlated.

to make the variable available to sub shells

    export a=1

    # spawn new shell
    bash

    echo $a
    # prints 1

note: executing a script, or a pipe will typically create a new shell
avoid sub shells if you don't need them

prevent string interpolation

    use single quotes

    script 'a$s'


escape single quote

    '"'"'
or
    '\''

    "don't"

    # use double quotes only on string
    'don'"'"'t'

    'don'\''t'


dynamic string arguments

The interpreter is smart enough to pass along a string that's already parsed, to the next command.  you do NOT need quotes around interpolated strings:

    example:

        # does not work ... too many single quotes
        file=.
        args=" -a '$file' "
        ls $args
        # fails.  looking for a file named \'.\'

        # works ... no single quotes within string:
        file=.
        args=" -a $file "
        ls $args

    summary: The only time you need to encapsulate string data in quotes is on the initial shell call (initial string parse). ... I think :)


add numbers

    num=$((num1 + num2))
    num=$(($num1 + $num2))       # also works
    num=$((num1 + 2 + 3))        # ...
    num=$[num1+num2]             # old, deprecated arithmetic expression syntax


error handling / exception handling

    # $? reports last error.
    # WARNING, IT IS CONSUMED ON FIRST TEST, SINCE EACH COMMAND WRITES TO IT

        grep 23412341  .
        grep: .: Is a directory

        $ echo $?
        2

        $ echo $?
        0

    # example test
    if [[ $? == 0 ]]
    then
        java Test
    fi


define / create an alias (bash syntax)

   
    alias name=value
    alias name2='command'
    alias name3='command arg1 arg2'
    alias name4='/path/to/script'
    alias name5='/path/to/script.pl arg1'


disable alias that overrides command, use ''

    'ls'

    unalias aliasname


alternate string quote syntax

c
    a = $'te\'st'

        strings that are scanned for ansi c like escape sequences.

i18n

    a = $"test"

    means get the international text. if there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign simply is ignored, which results in a normal double-quoted string.



copy var

    b=$a
    echo $b


if then, else if, else

    # test if variable is NOT set
    if [ -z "$VAR" ]; then

    # else if
    elif [ -z "$VAR2" ]; then

    else
   
   
    # end
    fi

    # note you can also write the 'then' on its own line


negation

    !     means not


test variable has a value

    if [ -z "$VAR" ];
    then
        echo "value is not set"
    fi


test file existence:


    if [ -f $FILE ];
    then
       echo "File $FILE exists."
    else
       echo "File $FILE does not exist."
    fi

    if [ ! -f $FILE ];
    then
       echo "File $FILE does not exist."
    fi


test if file contains pattern

    if grep -q FOOBAR "$File"; then
       # file has FOOBAR
    fi

test if file does not contain pattern

    if ! grep -q SomeString "$File"; then
       # Some Actions
    fi


then: can be written on the same line with a ; or on the next line

    # people likely do this because veritical space is limited on an 80x24 terminal
    if ! grep -q SomeString "$File"; then
        echo 1;
    fi

    # means the same as
    if ! grep -q SomeString "$File"
    then
        echo 1;

    fi


shell redirection

    cmd 2>      redirect stderr
    cmd &>      redirect both stderr and stdout

    cmd >&n     send output to descriptor n
    cmd m>&n    redirect output that would have gone to m to n
    cmd <&n     get input from descriptor n
    cmd <&-     close standard in


silence output


    # SILENCIO POR FAVOR!
    cmd &> /dev/null


pipes | are used to chain commands together.  output of the first command is wired up to the input of the next command (like snapping together legos)

    comand | tee file.txt   prints to screen and saves to file.txt


edit previous command

    ^foo^bar        executes previous command, except replaces foo with bar


loops:

    for loop

        for ((i=1;i<=100;i++));
        do
           # your-unix-command-here
           echo $i
        done
   
    while loop

        while ((1));
        do
           # your-unix-command-here
            sleep 1;
        done
       

    for

         for x in file1 file2 file3
         do
         sed 's/thier/their/g' $x > ,$x
         mv ,$x $x
         done
       

    # index iteration
     for ((i=0; i<10; i++)); do
         echo $i
     done


bash heredoc syntax

    # note no ; required
    # for quoting multiline strings
    cat << ENDOFBLOCK
            sdfg
ENDOFBLOCK

    Note: you can't assign this straight to a var
    For example, you can use a cat do

    variable=$(cat <<EOF
<html>
<body>
<p>THis is a test</p>
</body>
</html>
EOF
)


test loop

    You can preview what loops will do by putting an echo in front of the commands.  you don't want to realize there was a mistake after running the command a thousand times.

        # test
        for a in *
        do
            echo mv a b
        done

        # then use the command
        mv a b
   

    This can mess up " in output though, since echo will interpret it and discard.   if you need exact output, wrap the block in a literal heredoc.  THis will preserve all quotes " ' ` and meta chars

    So, intest you can use a heredoc to prevent interpolation

    cat <<DEBUG
    echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `
DEBUG
    # prints:
    #     echo PARAM=`  grep  "$ARG"  /var/tmp/setfile  | awk '{print $2}' `


piping output from one command to a loop

    find photos/ | while read file; do echo $file ; done;


loops in bash and variable scope:

  counter=0
  for f in $(find .)
  do
    let counter+=1
  done
  echo $counter

problem: if ls returns a large result, can overflow the buffer.

      counter=0
      find . | while read f
      do
        let counter+=1
      done
      echo $counter


    the problem: variables inside loop in their own shell, can't be used if needed.

    For read only access, use export in the outer context

        export var=1

    Though, this won't work if you need write access.  In this case you need < <() syntax
    for example


      counter=0
      while read f
      do
        let counter+=1
      done < <(find .)
      echo $counter

    ugly syntax, but works


handle whitespace in filename

    find <path> -mtime +7 | while read line
    do
        # just wrap with double quotes
        mv "$line" /somewhere/else
    done

you can embed a comment string in the heredoc marker

    # comment following line out after testing
    cat <<'#ENDDEBUG'

    echo PARAM=`ls -la  | awk '{print $3}' `
    #ENDDEBUG



parse delimted test with `cut`

    # list all users defined on system
    # split on :, print the first field
    cut -d: -f1 /etc/passwd


increment var

  y=$[y+1]
  y=$(($y+1))
  ((y++))


polling process (repeatedly testing)

    while (( 1 ));
     do   ls -lah sedOSEZA8 ;
     sleep 1;
    done


see all characters used in a file

    # change char to char\n
    cat Configuration | sed -e 's/\(.\)/\1\n/g' | sort | uniq



create a simple index page for .html files in a folder

 ls *.html -1 | while read f ; do echo "<a href='$f'>$f</a><br>"; done >> index.htm


scp - secure copy (use instead of ftp)

  scp  d_HomeBk.sql  user@server:/remote/path/to/dir/

  note: if there are spaces in file name use: encased/escaped quotes "\"  \""

    eg  

      scp  d_HomeBk.sql  user@server:"\"/path/to/file/\""

    or enclose entire string in single quotes, with double

      scp  d_HomeBk.sql  'user@server:"/path/to/file/"'

    use -r to copy recursively



find files

    # last 24 hours
    find . -mtime 0

    # older than a month
    find . -mtime +31

    # find files only
    find -type f


when backticks overflow command line, can group in blocks and pipe to a loop:

    % sh
    find . -type f -mtime -1 -print |
    fmt -1000 |
    while read files
    do pr -n $files
    done | lpr
    exit

    ...or use xargs
    xargs lpr < allfiles.tmp

or

    # NOTE the -I defines a string to replace
    # default behavior sends to first argument of command.
    command | xargs -I '{}' somecommand stuff '{}'
    command | xargs rm -rf


# copy a lot of image files, convert sizes


    ls -1 | while read file; do echo $file; \
    convert -resize 460 "$file" "/cygdrive/c/temp/yourfiles/$file" ; done


# slightly better complex ... can be stopped, restarted

    $destdir = /cygdrive/c/temp/yourfiles

    ls -1 | while read $file
    do
      if [[ ! -e $destdir/$file ]]
      then
        echo $file
        convert -resize 460 $file $destdir/$file
      fi
    done



internal file separator.  note: also there is the internal variable IFS (internal file seperator) that is noramally set to white space

    while can parse a line, such as

    IFS=':'
    cat  /etc/passwd | while read var1 var2 var3 var4
    do
     ...
    done

    # although awk is probably the better tool for this



the previous file edited on bash shell

    !$    





convert stream to lower case:

    alias lc="tr '[:upper:]' '[:lower:]'"
    alias uc="tr '[:lower:]' '[:upper:]'"
 
    eg:

     cat file.txt | lc



copy all but some files

    cpio can copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .snapshot (and anything in them).

    cd /source-dir
    find . -name .snapshot -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


    This command copies the contents of /source-dir to /dest-dir, but omits
    files  and directories named .svn (and anything in them).
    cd /source-dir
    find . -name .svn -prune -o \( \! -name *~ -print0 \)| cpio -pmd0 /dest-dir


profile vs bashrc

    profiles are for interactive login shells (that requre user/pass),
    rc is for non-interactive, non-login


mount cd rom manullly

     mkdir -p /mnt/cdrom && mount -t iso9660 -o ro /dev/cdrom /mnt/cdrom


functions

                # without arg, called like a command line tool
                f() { echo 1; };
                f;


                # with argument
                f() { echo $1; };
                f arg ;


floating point numbers

    bwa hahahah ... :-)
    bash doesn't support floating point arithmetic evaluation, but you can use the pretty-standard bc command in your function to help out.     example:

    function f_to_c { echo "5/9 * ($1 - 32)" | bc -l; }

    Though if your script is getting complex enough to use floating point numbers, it probably should be written in an actual scripting language (python, perl, ruby, php, etc) not bash.

arrays:

    If you are using arrays, you should start to wonder if you should be using bash at all and not another programming language.  :)
    But yes, you can do arrays.

    #simple array:

        name[index]=value

    # explicitly define numeric array


        declare -a chars  
        chars[0]="a"
        chars[1]="b"

       echo ${chars[0]}


    # set list of values

         declare -a arrayname=(element1 element2 element3)

    # bash shorthand: declare/assign array

        arr=(Hello World)

    # in referencing the value, you must enclose in {}

          echo ${arr[0]} ${arr[1]}

    To quote from the man page:
    The braces are required to avoid conflicts with pathname expansion.
    In addition the following funky constructs are available:

          ${arr[*]}         # All of the items in the array
          ${!arr[*]}        # All of the indexes in the array
          ${#arr[*]}        # Number of items in the array
          ${#arr[0]}        # Length of item zero

    # associative array or hash

        declare -A morse  
        morse[a]="dot; dash"
        morse[b]="dash; dot; dot; dot"
        morse[c]="dash; dot; dash; dot"
        # ...

       echo ${morse[$1]}   # write dots, dashes for letter

    # print all of array, use @ (index number) or * (all)

        echo ${morse[@]}

    # to also export declared vars

        declare -x ....


eval "string"

    execute code.  similar to `` or $(), but may read a bit easier

case:  (switch statement)

    Note: no break/exit needed

    # number of feet
    case $ANIMAL in

      horse | dog | cat) echo -n "four";;

      man | kangaroo ) echo -n "two";;

      alien ) echo -n "pi";;

      *)
        #default
        echo -n "an unknown number";;

    esac

print current user

    echo "$USER"
    id -u -n

    To print numeric UID, run:

    id -u

change passwd non-interactive

   
    # reads a list of users passwords (non interactive)
    # may be encrypted
    echo "username:newpass" | chpasswd


find all files not owned by user

    find '!' -uid 0
    find '!' -user root


can also  echo multiline strings

    echo "
    /var/log/top.log {
        missingok
    }
    " > /etc/logrotate.d/top
   

to avoid pipe in main loop

    this:

        while read log
        do
          echo $log
        done < <( svn log $1 | grep -E -e "^r[0-9]+ \| " | tac )

    is mostly the same as:

        svn log $1 | grep -E -e "^r[0-9]+ \| " | tac | while read log
        do
          echo $log
        done

ssh

    Warning: ssh reads from standard input, and can "eat" all your input if used in a while loop.
    Instead, in a loop use:

        cat /dev/null | ssh $u@$h  $COMMAND

    or
         ssh -n


    In general, you should close or redirect stdin for the other commands you
    run, to stop them reading from the file.

        sh call_other_script.sh </dev/null


rotate file based on size

    #!/bin/bash

    f=/var/log/top.log
    date         >> $f
    echo 'top:'  >> $f
    top -b -n 1  >>  $f

    echo '--'    >> $f
    echo 'ps:'   >> $f

    ps auxw      >>  $f

    echo '--'    >> $f
    echo ''      >> $f


    # don't let it get too big
    FILESIZE=$(stat -c%s "$f")
    MAXSIZE=10000000

    if [ $FILESIZE -gt $MAXSIZE ]; then
      mv -f $f $f.1
    fi



count word frequency

    tr ' ' '\n' < /tmp/kevtest_old.txt | sort | uniq -c | sort -nr | head

    27306
    5200 69
    1980 137
    1716 129
     728 93
     611 56
     611 55
     520 42
     520 41
     248 70

    alt:

        awk '{a[$1]++}END{for(k in a)print a[k],k}' RS=" |\n" file > myfile



test string for pattern


    build=123000
    if [[ $build =~ "3" ]]
    then
        echo "has 3"
    fi

    build=123000
    if [[ $build =~ "4" ]]
    then
        echo "has 4"
    fi


convert string list to array

    STRING="test1 test2 test3"

    # convert to array
    list=( $STRING )

    #Or more verbosely:

    declare -a list=( $STRING )

    PS: You can't export IFS and use the new value in the same command. You have to declare it first, then use it's effects in the following command:

    list=( first second third )
    echo "${list[*]}"


    #first element
    list=( first second third )
    echo "${list[0]}"
    # first


nohup


    # don't stop command if the terminal connction is lost

        nohup  one_command_only  &

    #execute multiple commands, call script in future

        # get both process ids
        function sleep_restart() {
                # to stagger node restarts, call script in future
                sleep=1
                echo "Will schedule start in $sleep seconds..."
                echo ""
                echo "    sleep $sleep && /yourcommand_restart"
                echo ""
                nohup sh -c "sleep $sleep && /yourcommand_restart >& /dev/null" &
                sleep 1
        }


find by permission

    find . -perm /222

    Search for files which are writable by somebody (their owner, or their group, or anybody else).

    find . -perm /220
    find . -perm /u+w,g+w
    find . -perm /u=w,g=w

    find . -perm /o=w


find loops in symlinks

    find . -follow -printf ""

find number of processes

    nproc
    cat /proc/cpuinfo

# count words or count repeated lines by occurance

    sort | uniq -c



remove XML tags, remove HTML tags

    sed -n '/^$/!{s/<[^>]*>//g;p;}'


base64 decode

    base64 --decode

    # eg, convert xml 64 data to jpeg
    curl  <YOUR_URL_HERE>  | sed -n '/^$/!{s/<[^>]*>//g;p;}' | base64 --decode > test.jpg


simple tail log, execute trigger when it reads a pattern

    # look for lines containing OutOfMemoryError
    log=/tmp/kevtest.txt
    pattern="OutOfMemoryError"
    while read line;
    do
        if [[ $line =~ $pattern  ]]
        then
            echo "hit $line !"
            break;
        fi
    done < <(tail -f $log)


    alt: with awk

    # no exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1");  }'

    # with exit
     tail -f /tmp/kevtest.txt | awk '/YourPatternHere/ { system("echo 1"); exit; }'


hex encoding to ascii

    d=`cat lib.php `

    echo -e $d  | awk '{printf "%s\n", $_}' > kevtest.php


test if socket port is open

    Use netcat (nc) for the client and server tests

    # basic interactive chat

        # server (use same ports on client and server)
        nc -l 7601 -v

        # client (use same ports on client and server)
        nc <HOST> 7601

        # note: exit with ctrl-C


    # scan port (non-interactive)

        nc -z <host> <port>


    # --
    # client examples
    # try socket connection
    # interactive

        nc  <host> <port>

    or (older)

        telnet  <host> <port>

    #scan (one shot, not interactive)

        nc -z <host> <port>

    # For a quick non-interative check (with a 5 seconds timeout):

        nc -z -w5 <host> <port>


    # --
    # SERVER
    # note: low number ports (such as 80) require higher perms
    # create socket that stays open indefinitely until manually closed
    #listen locally on port 1024 for all traffic and dump the output to STDOUT

        nc -l 1024 -k

    # more verbose

        nc -l 1024 -k  -v

    # close after first message

        nc -l 1024


    # more complex request (send mail)

        nc localhost 25 << EOF
        HELO host.example.com
        MAIL FROM: <user@host.example.com>
        RCPT TO: <user2@host.example.com>
        DATA
        Body of email.
        .
        QUIT
        EOF


--
BASH BRACKETS/BRACES/PARANS

There are a lot of brackets in Bash, like () (()) [] [[]], and some have overloaded meanings, which can get confusing.  Here's a quick overview of what they mean.  These types of symbols are hard to search for explanations as well.


bracket options in 'if' statements

    no brackets        execute shell commands and tests exit code
                    0 == true in this case, which can be counter-intuitive
                    'exit code' should really be named 'error code', so
                    'no errors' means true.  last code is stored in variable $?

    [ ] are simpler, older, POSIX compliant.
         it is usually just an alias to /bin/test, /bin/[

    [[ ]]  the double brackets are builting bash keywords

        in general use [[  vs [ in tests if you don't need to port to other shells

        it's newer and processes internally (better for whitespace handling)

        The double bracket enables additional functionality.
        For example, you can use && and || instead of -a and -o
        and there's a regular expression matching operator =~

    (( )) are used for arithemetic operations.  for example
            a=((a++))
            for ((i=0; i<10; i++)); do echo $i; done;


    $[ ]     deprecated syntax meaning (( ))


other bracket contexts


    ( )        1. are used to create a subshell, won't affect current shell
            ( cd /tmp; pwd ); pwd;

            2. are also used to create arrays, [ ] used for array index
                 array=(1 2 3)
                 echo ${array[1]}

            3. are also used used to define functions.

                # without arg, called like a command line tool
                f() { echo 1; }; f;
                # with argument
                f() { echo $1; }; f arg ;


            4. process substitution is:

                  cmd <(list)   is similar to        cat list | cmd
                or
                  cmd > >(list)    is similar to        cmd | list


                 The effect of process substitution is to make each list act like a file.

                however >() is more powerful than a normal pipe:
                You can't use | to redirect standard output and standard error to different programs.


    { }    does a couple things:
            1. unambiguously identifies variables. example ${a}${b}
            2. executes a sequence of commands in the CURRENT shell context, opposite of ()

                { date; top -b -n1 | head ; } >logfile

            There is a subtle syntactic difference with ( ), though (see bash reference) ; essentially, a semicolon ; after the last command within braces is a must,
            and the braces {, } must be surrounded by spaces.

    [ ] 1. can mean a range operator, or matching list

            ls [abc][1-4]
           
        2. can be the index designator in an array  ${a[0]}
               

    $( )     runs a command and drops the output in another.  like backticks ``
            usefule for nesting operations, example:

                # find string 'bar' in files named 'foo'
                grep bar $( find -name foo )




--
BASH STRING MANIPULATION

strip extension

    nameis=${filename%.php}
    nameis=${filename%.*}  # general


substring examples:

    The variable var contains /a/b/c/d/e.f.g:


        Expression   Result

        ${var}   /a/b/c/d/e.f.g

        # match first from left
        ${var#/*/}   b/c/d/e.f.g

        #match last from left
        ${var##/*/}   e.f.g

        # match first from right  
        ${var%.*}   /a/b/c/d/e.f

        # match last from right
        ${var%%.*}   /a/b/c/d/e

        # misc
        ${var%%/*/}   /a/b/c/d/e.f.g
        ${var%%/*}
        ${var%/b*}   /a
        ${var%%/b*}   /a


by length

    stringZ=abcABC123ABCabc
    #       0123456789.....
    #       0-based indexing.

    echo ${stringZ:0}                            # abcABC123ABCabc
    echo ${stringZ:1}                            # bcABC123ABCabc
    echo ${stringZ:7}                            # 23ABCabc

    echo ${stringZ:7:3}                          # 23A
                                                 # Three characters of substring.



    # Is it possible to index from the right end of the string?
       
    echo ${stringZ:-4}                           # abcABC123ABCabc
    # Defaults to full string, as in ${parameter:-default}.
    # However . . .

    echo ${stringZ:(-4)}                         # Cabc
    echo ${stringZ: -4}                          # Cabc
    # Now, it works.
    # Parentheses or added space "escape" the position parameter.



find replace (substring replacement)

    Replace first match of $substring with $replacement:

        ${string/substring/replacement}

    Replace all matches of $substring with $replacement.

        ${string//substring/replacement}




bash string editing operators

    Operator        Explanation

    ${variable#pattern}    Delete the shortest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable##pattern}    Delete the longest part of pattern that matches the beginning of variable's value. Return the rest.
    ${variable%pattern}    Delete the shortest part of pattern that matches the end of variable's value.Return the rest.
    ${variable%%pattern}    Delete the longest part of pattern that matches the end of variable's value.Return the rest.

    The patterns can be filename wildcard characters: *, ?, and []; with string editing operators, wildcards match strings in the same way they match filenames. (These are not sed-like regular expressions.) The first two operators, with #, edit variables from the front. The other two, with %, edit from the end. Here's a system for remembering which does what: you put a number sign (#) at the front of a number and a percent sign (%) at the end of a number.

Quick Language Reference: Perl

for executable scripts

    path to perl, first line can have hashbang
    #!/usr/bin/perl

    for portability, first line can have hashbang
    #!/bin/env perl

    a space after ! is ok
    #! /usr/bin/perl


test for errors:

    cat filename | perl -cw

show all lib paths

    perl -e 'print join "\n", @INC;'

look for installed libs

    perl -e 'print join "\n", @INC;' | while read f; do find $f -follow; done \
      | grep XML | sort | uniq

look for which libs are used

    #(all the use statments)
    grep -R -e '^use' .  | sed -e 's/^[^:]*://' -e 's/\s*;.*$//' \
      | sort | uniq

    #(just the lib names)
    grep -R -e '^use' .  | sed -e 's/^[^:]*://' -e 's/\s*;.*$//' \
      |  awk '{print $2}' | sort | uniq

show installed/cpan versions

    #!/usr/bin/perl
    # pass perl module name/regex as arg
    # eg:
    #     DBI /DBD::/

    use CPAN;

    printf("%-20s %10s %10s\n", "Module", "Installed", "CPAN");

    foreach $a (@ARGV) {
      foreach $mod (CPAN::Shell->expand("Module", $a)){
        printf("%-20s %10s %10s %s\n",
          $mod->id,
          $mod->inst_version eq "undef" || !defined($mod->inst_version)
            ? "-" : $mod->inst_version,
          $mod->cpan_version eq "undef" || !defined($mod->cpan_version)
            ? "-" : $mod->cpan_version,
          $mod->uptodate ? "" : "*"
        );
      }
    }


print library paths

    perl -e 'printf "%d %s\n", ++$i, $_ for @INC'

    #1 /wcs/lib/perl5/5.6.1/i686-linux-perlio
    #2 /wcs/lib/perl5/5.6.1
    #3 /wcs/lib/perl5/wcs/5.6.1/i686-linux-perlio
    #4 /wcs/lib/perl5/wcs/5.6.1
    #5 /wcs/lib/perl5/wcs
    #6 .


installation preference:

    1. try yum, if that doesn't work
    2. install with CPAN.  if that doesn't work
    3. compile manually (avoid if possible)

install perl libs with yum

    yum install
    #for example:
    yum install perl-Apache*


install perl with cpan

    perl -MCPAN -e 'install CGI::Builder'


manual installation

    # general process to compile c programs
    ./configure
    make
    make test
    make install


command line args

    # first argument:
    $ARGV[0]

file perms

    must have execute perms on script files

hello world

    print "hello\n";

http header:

    print "content-type: text/html \n\n";    #HTTP HEADER


require strict variable defs

    #!/bin/perl
    use strict;


perl variable markers

    $ for strings, literals
    @ for arrays
    % for hash (associative array)


strings

    "interprets $var variables"
    'literal syntax no interpolation'

string concat

    .

string length

 
    length()

remove new line

    chomp()



# perl heredoc syntax
$heredoc = <<END;
     Everything after the
     start of the
     here-doc is part of
     the string until we get
     to the
END

#  note end token can have NO leading whitespace
print $heredoc;

print line:

    print "line", "\n";

use data dumper for complex data, print to standard error log

    use Data::Dumper;
    print STDERR Data::Dumper::Dumper($node),"\n";

regex substitution

    # replace all words
    $sentence =~ s/london/London/g;

no match
 
    if ( $string !~ m/whatever/ ) {
        # it's not there
    }

regex match text, save to var

    # NOTE: $1 is a global var, may not be reset in a loop
    # () defines match

    # NOTE: =~ syntax seems backwards to me, but remember !~

    $string =~ m/whatever(sought_text)whatever2/;
    $soughtText = $1;


perl function, with variable

    # look ma, no args
    # everything is passed along the special @_ variable
    sub parse_layout {
      my $text = shift;

    }

    function rule of thumb:
        avoid long arg list.
        avoid referencing global data
        ideally, make all functions accept/return 1 hashref argument

if else

        if ( $line =~ /^#/ ) {
            # begins with comment;
        } elsif ( $line =~ /^\s*$/ ) {
            $endgroup = 1;
        } else {
            #data line;
        }

    comparison

        ==  != > < <= >=    for numbers only!
        eq, ne                for strings only!


    operators

        &&    and
        ||    or
        !    not

perl split string

    my $info = "part:part:part:part";
    my @parts = split(/:/, $info);

remove trailing newline, if it's there

    chomp();

trim

    sub trim {
        my $s = shift;
        $s =~ s/^\s+//;
        $s =~ s/\s+$//;
        return $s;
    }


define array/hash

    # actual named variables
    my @coins = ("Quarter","Dime","Nickel");
    my %account = ('username' => 'jimmyboy', 'password' => 's3cret!');

    # !!!! WARNING !!!!

    # perl doesn't really have a low-level concept of array like you might expect
    # for any complex array manipulation, use array/hash *pointers* instead (below)
    # unlike arrays, these can be assigned to another variable
    # or redefined in a loop
    # the pointers will work more like the array/hash in other languages

    NOTE: perl array "weirdness" as compared to php
    http://www.webreference.com/programming/perl/nested/index.html
    http://docstore.mik.ua/orelly/perl4/prog/ch09_01.htm


define array/hash pointers (what you should use)

    arrays are handled a bit differently than in php
    arrays loop by ref by default


    array reference

        my $array = [];

    hash reference

        my $hash = {};



    create from existing array/hash


        $aref = \@array;         # $aref now holds a reference to @array
        $href = \%hash;          # $href now holds a reference to %hash
        $sref = \$scalar;        # $sref now holds a reference to $scalar




define nested or complex data structure:
    grok:
        somewhat like javascript/python with {} []
        like php with =>

        my $VAR1 = {
                  'alpha' => [
                               123,
                               456
                             ],
                  'beta' => 'x'
                };


access array/hash pointer

    use $href-> to dereference


    # like c, can dereference pointer properties with ->


        $hashref->{key}
        $hashref->{'key'}   #quotes on keys optional
        $arrayref->[0]

    # alt syntax: use double $$ ... uglier

         $$hashref{'asdf'}
         $$arrayref[0]

dereference array ref to array

     @$ret

dereference array ref to hash
     %$ret

dereference field of a class:

    @{$user->{'Accounts'}}
    %{$user->{'Accounts'}}


loop over array

    foreach my $part ( @parts ) {
        print $part;
    }

loop over keys

    foreach my $key (  keys %hash ) {
        print $key;
    }


    NOTE: build a list of items, then perform operation on the list


passing refernce

    my %some_hash = ();
    return \%some_hash
 
    my %some_array ...
    return \@some_hash


sorting

    http://raleigh.pm.org/sorting.html

    case insensitive

        @s = sort {lc $a cmp lc $b} @a;

    element length

        @s = sort {lenth $a <=> length $b} @a;

    sub routine

        @s = sort mycriteria @a;
           sub mycriteria {
             my($aa) = $a =~ /(\d+)/;
             my($bb) = $b =~ /(\d+)/;

             sin($aa) <=> sin($bb) ||
             $aa*$aa <=> $bb*$bb;
           }


# two levels of sorting
# $a and $b are predefined

    sub inorder {
      $ACCT_SORTORDER{$user->{$a}{'TypeID'}} <=> $ACCT_SORTORDER{$user->{$b}{'TypeID'}}
        or
      $user->{$a}{'Nickname'} cmp $user->{$b}{'Nickname'};
    }

    my(@accounts) = sort inorder @{$user->{'Accounts'}};


try/catch exception

    here's the version of perl's try/catch

      eval {
        $response_doc = $parser->parse_string( $xml_str );
      };
      # catch parse exception            
      if($@) {
        # error
      }

      # reset $@ if calling again, this is global var
   

system call
    system "curl --basic --user $username:$password --data-ascii status='$message' http://twitter.com/statuses/update.json";
 
    # with backticks
    my $ret =
    `curl --basic --user $username:$password --data-ascii status='$message' http://twitter.com/statuses/update.json`;




file handling:


    mode    operand    create    truncate
    read    <      
    write    >        Y    Y
    append    >>        Y  
    Each of the above modes can also be prefixed with the + character to allow for simultaneous reading and writing.

    mode    operand    create    truncate
    read/write    +<      
    read/write    +>    Y    Y
    read/append    +>>    Y  


    # open file  for read
    open FILE, "
    As you might have guessed already if you just want read access you can skip the mode just as we did in the very first example above.

    Reading files

    If you want to read a text file line-by-line then you can do it as such:

    my @lines =
    The
    while (      print $_;
    }

    The $_ variable is automatically set for you to the contents of the current line. If you wish you may name your line variable instead:

    while (my $line =

    # release handle
    close FILE;


    # write to file
    open FILE, ">file.txt" or die $!;
    print FILE $str;
    close FILE;


perl oop

OOP perl is a little like Python

for more info:

    http://www.tutorialspoint.com/perl/perl_oo_perl.htm


    #!/usr/bin/perl

    package Person;

    sub new
    {
        my $class = shift;
        my $self = {
            _firstName => shift,
            _lastName  => shift,
            _ssn       => shift,
        };
        # Print all the values just for clarification.
        print "First Name is $self->{_firstName}\n";
        print "Last Name is $self->{_lastName}\n";
        print "SSN is $self->{_ssn}\n";
        bless $self, $class;
        return $self;
    }
    sub setFirstName {
        my ( $self, $firstName ) = @_;
        $self->{_firstName} = $firstName if defined($firstName);
        return $self->{_firstName};
    }

    sub getFirstName {
        my( $self ) = @_;
        return $self->{_firstName};
    }
    1;



# using
# use the source file, and call

    use Person;

    $object = new Person( "Mohammad", "Saleem", 23234345);
    # Get first name which is set using constructor.
    $firstName = $object->getFirstName();


# inherit

    our @ISA = qw(Person);    # inherits from Person



# CGI
#query parameters
# get/post parameters

    http://perldoc.perl.org/CGI.html


    use CGI;
    my $q = CGI->new;

        # Process an HTTP request
     @values  = $q->param('form_field');
     $fh      = $q->upload('file_field');
     $riddle  = $query->cookie('riddle_name');
     %answers = $query->cookie('answers');
    # Prepare various HTTP responses

    print $q->header();
    print $q->header('application/json');

    $cookie1 = $q->cookie(-name=>'riddle_name', -value=>"The Sphynx's Question");
    $cookie2 = $q->cookie(-name=>'answers', -value=>\%answers);
    print $q->header(
        -type    => 'image/gif',
        -expires => '+3d',
        -cookie  => [$cookie1,$cookie2]
        );
   print  $q->redirect('http://yoursite.com/your/page');

Quick Language Reference: Ruby

# interactive ruby shell:

    # install on linux
    yum -y install ruby ruby-irb

    # install on windows
    use cygwin, install all ruby packages.  includes gem and irb

    # To quit IRB, type quit, exit or just hit Control-D.

    # load file
    # or put ruby code in  ~/.irbrc
    irb -r 'file'

    # dynamically load file
    irb
    require 'file'


# equiv of print_r, var_dump (think "pretty print")

    pp object

# hash tag

    #!/usr/bin/env ruby

# comments

    #

# define/set variable

    a = 1
    b = "some string"

    c = nil

#print

    puts "hi"

# read line

    line = gets


# define method/function

    def hi(name)
        puts "hi #{name}!"
    end


# call function
    hi
    hi("name")

# introspection: print all methods of object

    require 'pp'
    pp 5.methods



# default values, functions in string

    def hi(name = "World")
        puts "Hello #{name.capitalize}!"
    end


# simple class

    class Greeter

      # constructor
      def initialize(name = "World")
        # use meta char @ to access class properties
        @name = name
      end

      def say_hi
        puts "Hi #{@name}!"
      end

      def say_bye
        puts "Bye #{@name}, come back soon."
      end

    end


# instantiate class with .new

    g = Greeter.new("John Smith")
    g.say_hi
    g.say_bye


# list public methods for object:

    # include all ancestors
    Greeter.instance_methods

    # only in immediate class
    Greeter.instance_methods(false)


# test if a method is defined for the object

    # is name defined on the object?
    g.respond_to?("name")


# alter existing class

    #NOTE: The changes will be present in any new objects you create and even available in existing objects of that class.

    class Greeter

        # public getter and setter  
       attr_accessor :name

        # only getter
       attr_reader :thing

    end


# array

    names = ["Albert", "Brenda", "Charles", "Dave", "Englebert"]

     
    # size:
    names.size  
 

# loop over array

    # pipes into bound variable 'name'
     names.each do |name|
 
    puts "Hello #{name}!"
     end

# join

    names.join(", ")


# hash

    H = Hash["a" => 100, "b" => 200]

    puts "#{H['a']}"
    puts "#{H['b']}"

    #This will produce following result:

    #100
    #200


# get hash keys as array

    keys = H.keys
    puts "#{keys}"

# loop over hash

    hash.each do |key, array|
      puts "#{key}-----"
      puts array
    end

    hash.keys.sort.each do |key|
      puts "#{key}-----"
      hash[key].each { |val| puts val }
    end


    puts h.sort.map {|k,v| ["#{k}----"] + v}


# hash methods:

        http://www.tutorialspoint.com/ruby/ruby_hashes.htm


# duck typing.  you can probe capabilities of object

    class Greeter2
        def say_bye
          if @names.nil?
            puts "..."
          elsif @names.respond_to?("join")
            # Join the list elements with commas
            puts "Goodbye #{@names.join(", ")}. "
          else
            puts "Goodbye #{@names}. "
          end
        end
    end


# test if class file is called directly

    if __FILE__ == $0
        # I am filename  

    end


# parse standard input stdin

    #!/usr/bin/env ruby

    chars = 0
    words = 0
    lines = 0

    # Count the stuff

    while line = gets
      chars += line.size
      words += line.split.size
      lines += 1
    end

    # Display the results

    puts "Chars=%s, Words=%s, Lines=%s" % [
      chars, words, lines
    ]


# read all lines

    lines = readlines
    puts "Chars=%d, Words=%d, Lines=%d" % [lines.join.size, lines.join.split.size, lines.size]


# code blocks

    Procs are objects that can be called.
    Sending the call method to the proc object executes the code block.
    The [] operator is a synonym for call.


    {
        |boundvariable|
    }


    do |var|

    end


    block = proc { |x| puts "Called with #{x}" }

    # calling
    block.call(1)
    block.call(2)

    block[222]


# using blocks (notes from Matz and the interwebs)

    Matz: Blocks are basically nameless functions. You may be familiar with the lambda from other languages like Lisp or Python. Basically, you can pass a nameless function to another function, and then that function can invoke the passed-in nameless function. For example, a function could perform iteration by passing one item at a time to the nameless function. This is a common style, called higher order function style, among languages that can handle functions as first class objects. Lisp does it. Python does it.  Even C does it with function pointers. Many other languages do this style of programming. In Ruby, the difference is mainly a different kind of syntax for higher order functions. In other languages, you have to specify explicitly that a function can accept another function as an argument. But in Ruby, any method can be called with a block as an implicit argument. Inside the method, you can call the block using the yield keyword with a value.

    SO: Blocks are not objects, they are syntactic structures; this is why they cannot be assigned to a variable. This is a privilege reserved for objects.
    Blocks are closures -- they're functions that close over the enclosing scope. They don't conceptually "belong to" a given object.

    Blocks that directly follow a method call are implicitly passed to the method.
    The method may call the proc by invoking yield and passing any arguments.
    You can test for the presence of a block using block_given?.
    yield(arg) if block_given?


    def three_times(arg)
      yield(arg)
      yield(arg)
      yield(arg)
    end

    three_times("x") { |val| puts val }
    three_times(22) do |thing|
      puts "Got #{thing}"
    end

    # output

    x
    x
    x
    Got 22
    Got 22
    Got 22


    The block may be explicitly passed using the & operator.
    def three_times(arg, &block)
      block.call(arg)
      block.call(arg)
      block.call(arg)
    end


# lambda vs procs

    There are two slight differences between lambda and Proc.new.

    First, argument checking. The Ruby documentation for lambda states: Equivalent to Proc.new, except the resulting Proc objects check the number of parameters passed when called.. Here is an example to demonstrate this:

    Second, there is a difference in the way returns are handled from the Proc. A return from Proc.new returns from the enclosing method (acting just like a return from a block.


    def try_ret_procnew
        ret = Proc.new { return "Baaam" }
        ret.call
        "This is not reached"
    end

    # prints "Baaam"
    puts try_ret_procnew


    While return from lambda acts more conventionally, returning to its caller:

    def try_ret_lambda
        ret = lambda { return "Baaam" }
        ret.call
        "This is printed"
    end

    # prints "This is printed"
    puts try_ret_lambda


    I would probably recommend using lambda instead of Proc.new when possible.


#  unit testing

    require 'test/unit'
    require 'trio'

    class TestTrio < Test::Unit::TestCase
      def setup
        @trio = Trio.new(1,2,3)
      end

      def test_trio
        assert_equal [1,2,3], @trio.to_a
      end
    end


# database io

    require 'dbi'

    db = DBI.connect("DBI:Pg:poc", 'jim', nil)
    sql = %{
      SELECT name, id
      FROM users
      WHERE id < 4
    }
    db.select_all(sql) do |row|
      puts "User #{row['name']}, id #{row['id']}"
    end
    db.disconnect


# composite pattern, loops

    << is the polymorphic append operator.
    { ... } is a code block. It is automatically passed to the each function.
    |installer| is an argument to the code block.
    list.each { |item| code } is equivalent to the for loop
    do ... end is also code block.


    class MasterInstaller
      def initialize
        @installers = []
      end

      def register(installer)
        @installers << installer
      end

      def install
        for installer in @installers
          installer.install
        end
      end

      def uninstall
        @installers.each { |installer|
          installer.uninstall
        }
      end

      def backup
        @installers.each do |installer|
          installer.backup
        end
      end
    end


# what's the ?

    ? as a method name suffix (e.g. include?) indicates that the method returns a true/false value.

# includes  (load module)

    require: finds the file and loads it (once)
 
    new: sent to a Class object will cause a new instance to be created. Parameters passed to new will automatically be passed to initialize.
 
    Parenthesis are optional if the result is not ambiguous.
 
    ARGV is a list of command line arguments
 
    No interface declaration is needed for installers. As long as they implement the needed methods, the master installer can use them.
 

    # find file, load
    require 'masterinstaller'
    require 'fileinstaller'
    require 'sqlinstaller'

    m = MasterInstaller.new

    m.register(FileInstaller.new("xyz.rb", "bin"))
    m.register SqlInstaller.new(
      "INSERT INTO a_table VALUES(1,2,3);" )

    commands = ['install', 'uninstall', 'backup']
    ARGV.each do |command|
      puts "Command is #{command}"
      m.send(command) if commands.include? command
      puts
    end


# reserved words

 
    BEGIN        do        next    then
    END            else    nill    true
    alias        elsif    not        undef
    and            end        or        unless
    begin        ensure    redo    until
    break        false    rescue    when
    case        for        retry    while
    class        if        return    while
    def            in        self    __FILE__
    defined?    module    super    __LINE__


# gotchas:

 
    No Primitives. Integers and floats are objects
    -1.abs => 1

    No Semi-colons
    nil, not null

    nil is an object
    nil.nil? => true

    nil and false are false
    everything else, including 0, is true

    Picky about spaces
    f (arg) gives warning.
    f(arg) is ok.


    Different type model
    Duck Typing!
    No interfaces needed.

    Expression oriented syntax.
    Almost everything returns a value
    x = if a==0 then 5 else 1 end
    Methods automatically return their last expression.

    Single Inheritance
    But mixins are available

    Per-object methods are allowed (singleton methods)
    Classes are always open (even built in classes)


# messages

    OO in Java is defined in terms of calling functions
    Member functions, static functions, etc.
    OO in Ruby is defined in terms of sending messages
    Sending messages, obj.send(:method_name)

    class Proxy
      def initialize(target)
        @target = target
      end
      def method_missing(sym, *args, &block)
        @target.send(sym, *args, &block)
      end
    end

    class Dog
      def talk; puts "WOOF"; end
    end

    d = Dog.new
    p = Proxy.new(d)

    p.talk

# file

Here, the File.open method takes a block. It then opens a new file (in "append" mode), and yields the open file into the block. When the block completes, Ruby closes the file. Except that Ruby doesn't just close the file when the block completes; it guarantees that the File will be closed, even if executing the block results in a raise. Let's take a look at the implementation of File in Rubinius:

    def append(location, data)
      path = Pathname.new(location)
      raise "Location does not exist" unless path.exist?
   
      File.open(path, "a") do |file|
        file.puts YAML.dump(data)
      end
   
      return data
    end

# modules are called "gems" or "ruby-gems"

    # install with yum
    yum install rubygem-json


# introspection

    Depending on what information you're looking for, try:

    obj.methods
    and if you want just the methods defined for obj (as opposed to getting methods on Object as well)

    obj.methods - Object.methods
    Also interesting is doing stuff like:

    obj.methods.grep /to_/
    To get instance variables, do this:

    obj.instance_variables
    and for class variables:

    obj.class_variables


# list all installed gems

    gem list

# list all remote gems

    gem list rhc --remote --all

Quick Langauge Reference: Python

interactive prompt, for quickly testing code

    run (with no args):

        python


    # this starts an interactive shell  
    print "hi"

    use:
        ctrl-c to stop execution
        ctrl-d to exit


introspection/reflection

    # will print out object methods in the shell
    dir()

    #example

    import struct
    dir()
    dir(struct)  


line comment

    #

mulitline comment

    """
    you can use pydoc style quotes,
    though it's kinda a hack
    """

continue long lines

    this is a long line \
    that will be continued

combine two lines

    print "hi" ; print "there"


tab/spaces:

    best practice is (apparently) to use spaces instead of tabs

Note: This seems counter intuitive to me. No one seems to agree on how many spaces to use.   PEP 8 recommends 4 spaces, and Google recommends 2 spaces.  Personally, I think it makes more sense to use \t and let everyone set their own tabstop.


print

    print "hello world"


    concatinate string

    # with space
    a = "hello",  "world"
    print a
    # hello world


    # without space
    print "hello" + "world"
    helloworld


    WARNING: python3 has apparently changed this basic function syntax.  :-)
    You'll need to use parens with python3: print(string)


variables, no declaration needed

    # simple
    a = 1

    # can assign a list to a list
    a,b,c = 1,2,3

    #can chain
    a = b = c = 1


python assumes all function variables are local.  to access the global scope use

    global varname


basic strings

    s = 'A string constant'
    s = "another constant"



There is no difference between ' and " (unlike in Perl and PHP).  Typically is seems like python code I see tends to prefer ' (easier to type?)

multi line strings use a triple-quote syntax:

    s = """
    This is
    your string
    """


analogue of php require_once (include)


    EXAMPLE

        #### (file: config.py)
        # can be nested data structure

        constant1 = "value1"
        constant2 = "value2"


        ####  (consumer.py)
        # note, no .py extension.

        import config

        # note, by default, everything is under the namespace:
        print config.constant1
        print config.constant2


    better - wrap with try/except

        # create file named modulename.py
        # then in second script run  
        try:
            import modulename
        except ImportError:
            print 'importing modulename failed'



command line args

    import sys

    # 1 is first arg
    a = sys.argv[1]



function

    def concat(a, b):
        return a + b;


    functions can be called with named arguments

        concat('1', '2')

    or

        concat( a = '1', b = '2' )


    functions can also set default values

    def example( a, b = 0 ):
        return a + b;



nested functions

    you can define function in a function and return it:


    def example():
        def ret():
            print "hi"
        return ret

    t = example()
    t()
    #hi


more on strings

    a = 'string'
    a = "string"
    a = """string"""
    a = '''string'''

    # unicode
    a = u'strng'

    # binary
    a = b'strng'

    # regular expression
    regex = r"(\w)"


string formatting

    WARN: python3 changes this syntax:

    a = 5
    b = "hello"

    buf = "A = %d\n , B = %s\n" % (a, b)
    print buf

    # prints:
    # A = 5
    #  , B = hello

    c = 10
    buf = "C = %d\n" % c
    print buf
    C = 10



    You could as well use format:

    >>> print "This is the {}th tome of {}".format(5, "knowledge")
    This is the 5th tome of knowledge


    note: % will be deprecated in python 3.1


# string manipulation

    # to lower  case
    a.lower()

    # replace
    a.replace( find, start, [num_of_times] )

    # starts with string
    a.startswith( "/" )

    # ends with string
    a.startswith( "/" )


# regular expressions

    import re

    # substitute
    re.sub(r"(\w)(\w+)(\w)", yourreplacement, yourstr)

    # match
    if re.search('http', yourstr) != None:

int

    b = 2

float

    b = 2.0

booleans

    True, False

null value

    None


conditional

    if a < b:
        do_something();
    elif a < c:
        do_something_else()
    else:
        default_case()


logical operators (these do what you'd expect)

    and
    or
    not

    # don't forget and type &&  ||  !  :-)


main hook

    if __name__ == "__main__":
        main()


exit program:


    import sys
    sys.exit(2)


    #another example
    if sys.argv[1] == '':
        print "usage is ./yourprogram.py site url "
        sys.exit(2)



array

    empty list
    a = []

    with values
    a = ['spam', 'eggs', 100, 1234]

    # length

    len(a)

    #note: multidimensional arrays work as you'd expect

    # whitespace between elements is ignored:
    # valid syntax:
    a = [1,
            2,
            3,
            4
                ]


test if value is in array

    has_spam = 'spam' in s


loops

    for n in [1,2,3]:
        print n


list/array methods:

    list.append(item)
    list.extend(List)
    list.insert(i, item)
    list.remove(item)
    list.pop(i)
    list.index(item)
    list.count(item)
    list.sort()  # in place
    list.reverse()


hash/dictionary


    stock = {
        "key": "value",
        "key2": 2
    }

    stock['key3'] = 2

    # remove item

    del stock['key3']


nested data structure

    # nest arrays and
    a = [ {'a': 1}, {'a': 2} ]

    print(a[1]['a'])
    #prints: 2

loop over dictionary/hash

    d = {'x': 1, 'y': 2, 'z': 3}
    # loop over keys, read into var 'key'
    for key in d:
        print key, 'corresponds to', d[key]

    # loop over key and value
    for key, value in d.iteritems():
        print key, 'corresponds to', value


    breaking out of a loop:

        break        exists loop
        continue    skips iteration


cast

    int()
    float()
    str()
    chr()
    unichr()
    ord()
    hex()
    oct()

tuple (invariant data, like a constant struct)

    coord = (2,3)


set (like a dictionary with no data)

    a = { 'a', 'b' }

while loop

    while link != '':
        # do something

files

    f = open('doo.txt')
    while line = f.readLine()
        print line
    f.close()

user input

    name = raw_input("enter your name:")

array slice

    select all items j <= k < j
    a[i:j]

    -1 counts starting from end of list

    replace list subset with another

    a[1:2] = [1,2,3,4]

    concatenate lists
    a = a + b


tuples

    pack data together as a single unit
    a = ('GOOG', 100, 635.2)

    access like array

set

    like array keys, no duplicates

    s = set([3,4,5])


    operations
    |    union
    &    intersection
    -    difference (subtract)
    ^    nand


place holder

    if you need to stick something in a code block for syntax check
    does nothing but fill space

        if 1:
            pass

test if string contains substring

    x = 'hello'
    y = 'll'
    y in x
    # True

index of string

    >>> print "hello".find('h')
    0
    >>> print "hello".find('ss')
    -1


exceptions

    try:
        do_somthing()

    except:
        print "error: your error "



    full syntax
    except NameError, IndexError as msg:
        print "error", msg      


    raising execptions (throw)

        raise ValueError('invalid number')

    print error # older syntax

        try:
            doX()
        catch Exception, e:
            print e

generator

    use instead of return
        yield

    the function maintains persistent state and pauses at the yeild line

    to grok: it's a static function with static global variables.


 
lamba

    for short anon function, no (), no return statement, one line
 
        square = lambda x: x ** 2

        print square(2)
 


Object Oriented Programming in Python

For me, OOP in Python is difficult to use compared to Java/C#.  To me it just feels like OOP was bolted onto Python as an after-thought.  Or probably more accurately, it copied from a language which bolted OOP on as an after-thought.  OOP was not present in the pre 1.0 releases of Python.

Overall, I really wish Python had (a) explicit variable declarations like `var x` and (b) implicit self.  The scoping rules also feel screwy (why do we need `global`?  If can see a variable in a parent scope, why can't Python?)

Otherwise, you might initially think if you write a set of functions that access global variables, all you need to do to convert them into a class, is to indent them all and add a class name to the top.  For example, this syntax would make sense to me:


    # i'm a function accessing a global variable
    a = 1
    def get_a():
        return a

    # now i'm a class
    class your_class:
        a = 1 # now an instance variable
        def get_a():
            return a

This is simple and makes sense to me.  Unfortunately it won't work at all. :-)  Python forces you to rewrite every single function signature and variable reference.

After struggling with Python's approach, IMO the easiest way to understand understand OOP in Python is to realize that: Python is NOT really an OOP language, any more than Perl is.  It's a multi-paradigm language which happens to provide a little OOP "syntax sugar."  But it's really bitter tasting sugar. :-)


As a result, the OOP "object.function" syntax is largely redundant.  It's passing an object reference around as the first argument, and has a very lightweight syntax manipulation that just converts this syntax:

 
    obj.f(a)  ->  f(obj, a)

IOW, it's just moving the "obj" to inside the function.  That's not OOP; that's just a regular expression.


Anyway, accepting the screwy OOP syntax, here's an object example:

    class Test:

        # instance variable  
        your_variable = ''

        # static variable
        your_static = ''

        # constructor
        def __init__(self, a):
            # no class scope of variables.  must use explicit self.
            self.your_variable = a  
            Test.your_static = a  

        # like perl class
        def your_method( self, a ):
            self.your_variable = a  
            # ...


initialize class

    # NOTE: no 'new' keyword
    # this is less painful

    t = Test()


extending another class

    class Test( BaseClass ):

        #...

execute shell command

    import subprocess

    p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    for line in p.stdout.readlines():
        print line,
    retval = p.wait()



    older method (prints output)

        os.system(cmd_string)



modules:

    save as yourfile.py


        import yourfile

    call as:

        yourfile.function()


    import indivdual functions into script namespace


        from yourfile import *
        from yourfile import f1, f2


urllib vs urllib2


    generally you should use the newest lib 2
    urlib (old) has the encode function

     urllib.urlencode({'abc':'d f', 'def': '-!2'})
    'abc=d+f&def=-%212'


    in python 3, everything has been merged into urllib



easy install for modules

    # shell level installer

    wget http://peak.telecommunity.com/dist/ez_setup.py
    python ez_setup.py


    # cache downloade files with
    easy_install  -b  /path/to/build/dir/

    # install from cache
    easy_install  /path/to/build/dir/


install pip to install new modules. like PCAN for Pearl


    new versions of python (>3.4) already include pip

    for old versions download

        https://bootstrap.pypa.io/get-pip.py

    run

        python    get-pip.py

    add python Scripts/ to PATH, if it's not already


    #verbose output
    pip -vvv install flask


better parsing of command line args

    # allow a hook for cli
    if __name__ == "__main__":
        opt = OptionParser()
        opt.add_option("-d", "--database", help="EZCONNECT string", action="store",
                     type="string", dest="db")
        opt.add_option("-n", "--requests", help="number of requests", action="store",
                     type="int", dest="requests", default=10)
        opt.add_option("-c", "--concurrency", help="number of concurrent connections",
                     action="store", type="int", dest="concurrency", default=1)
        opt.add_option("-s", "--sql", help="SQL query or PL/SQL block",
                     action="store", type="string", dest="sql")
        opt.add_option("-b", "--bind", help="dictionary of bind parameters",
                     action="store", type="string", dest="bind")
        (options, args) = opt.parse_args()
        bench = Orabench(options)
        bench.run()


python debugger

    within a python file:

        import pdb
        ...
        pdb.set_trace() begin the debugger at this line when the file is run normally

    start debug mode

        python -m pdb  <name>.py  [args]    begin the debugger
        help [command] view a list of commands, or view help for a specific command

    control execution

        l(ist)       list 11 lines surrounding the current line
        w(here)      display the file and line number of the current line
        n(ext)       execute the current line
        s(tep)       step into functions called at the current line
        r(eturn)     execute until the current functionṡs return is encountered
        b[#]         create a breakpoint at line [#]
        b            list breakpoints and their indices
        c(ontinue)   execute until a breakpoint is encountered
        clear[#]     clear breakpoint of index [#]
        q(uit)       exit the debugger

    changing variables / interacting with code

        p <name>      print value of the variable <name>
        ! <expr>      execute the expression <expr>



operators

    Operator    Description    Example

    +    Addition - Adds values on either side of the operator     a + b will give 30
    -    Subtraction - Subtracts right hand operand from left hand operand     a - b will give -10
    *    Multiplication - Multiplies values on either side of the operator     a * b will give 200
    /    Division - Divides left hand operand by right hand operand     b / a will give 2
    %    Modulus - Divides left hand operand by right hand operand and returns remainder     b % a will give 0
    **    Exponent - Performs exponential (power) calculation on operators     a**b will give 10 to the power 20
    //    Floor Division - The division of operands where the result is the quotient in which the digits after the decimal point are removed.     9//2 is equal to 4 and 9.0//2.0 is equal to 4.0



    ==     Checks if the value of two operands are equal or not, if yes then condition becomes true.     (a == b) is not true.
    !=     Checks if the value of two operands are equal or not, if values are not equal then condition becomes true.     (a != b) is true.
    <>     Checks if the value of two operands are equal or not, if values are not equal then condition becomes true.     (a <> b) is true. This is similar to != operator.
    >     Checks if the value of left operand is greater than the value of right operand, if yes then condition becomes true.     (a > b) is not true.
    <     Checks if the value of left operand is less than the value of right operand, if yes then condition becomes true.     (a < b) is true.
    >=     Checks if the value of left operand is greater than or equal to the value of right operand, if yes then condition becomes true.     (a >= b) is not true.
    <=     Checks if the value of left operand is less than or equal to the value of right operand, if yes then condition becomes true.     (a <= b) is true.


    =    Simple assignment operator, Assigns values from right side operands to left side operand     c = a + b will assigne value of a + b into c
    +=    Add AND assignment operator, It adds right operand to the left operand and assign the result to left operand     c += a is equivalent to c = c + a
    -=    Subtract AND assignment operator, It subtracts right operand from the left operand and assign the result to left operand     c -= a is equivalent to c = c - a
    *=    Multiply AND assignment operator, It multiplies right operand with the left operand and assign the result to left operand     c *= a is equivalent to c = c * a
    /=    Divide AND assignment operator, It divides left operand with the right operand and assign the result to left operand     c /= a is equivalent to c = c / a
    %=    Modulus AND assignment operator, It takes modulus using two operands and assign the result to left operand     c %= a is equivalent to c = c % a
    **=    Exponent AND assignment operator, Performs exponential (power) calculation on operators and assign value to the left operand     c **= a is equivalent to c = c ** a
    //=    Floor Dividion and assigns a value, Performs floor division on operators and assign value to the left operand     c //= a is equivalent to c = c // a


    &     Binary AND Operator copies a bit to the result if it exists in both operands.     (a & b) will give 12 which is 0000 1100
    |     Binary OR Operator copies a bit if it exists in eather operand.     (a | b) will give 61 which is 0011 1101
    ^     Binary XOR Operator copies the bit if it is set in one operand but not both.     (a ^ b) will give 49 which is 0011 0001
    ~     Binary Ones Complement Operator is unary and has the efect of 'flipping' bits.     (~a ) will give -61 which is 1100 0011 in 2's complement form due to a signed binary number.
    <<     Binary Left Shift Operator. The left operands value is moved left by the number of bits specified by the right operand.     a << 2 will give 240 which is 1111 0000
    >>     Binary Right Shift Operator. The left operands value is moved right by the number of bits specified by the right operand.     a >> 2 will give 15 which is 0000 1111


    and     Called Logical AND operator. If both the operands are true then then condition becomes true.     (a and b) is true.
    or    Called Logical OR Operator. If any of the two operands are non zero then then condition becomes true.     (a or b) is true.
    not    Called Logical NOT Operator. Use to reverses the logical state of its operand. If a condition is true then Logical NOT operator will make false.     not(a and b) is false.



    in    Evaluates to true if it finds a variable in the specified sequence and false otherwise.     x in y, here in results in a 1 if x is a member of sequence y.
    not in    Evaluates to true if it does not finds a variable in the specified sequence and false otherwise.    x not in y, here not in results in a 1 if x is not a member of sequence y.


    is    Evaluates to true if the variables on either side of the operator point to the same object and false otherwise.     x is y, here is results in 1 if id(x) equals id(y).
    is not    Evaluates to false if the variables on either side of the operator point to the same object and true otherwise.     x is not y, here is not results in 1 if id(x) is not equal to id(y).



Python Operators Precedence. The following lists all operators from highest precedence to lowest.


    Operator    Description
    **    Exponentiation (raise to the power)
    ~ + -    Ccomplement, unary plus and minus (method names for the last two are +@ and -@)
    * / % //    Multiply, divide, modulo and floor division
    + -    Addition and subtraction
    >> <<    Right and left bitwise shift
    &    Bitwise 'AND'
    ^ |    Bitwise exclusive `OR' and regular `OR'
    <= < > >=    Comparison operators
    <> == !=    Equality operators
    = %= /= //= -= += *= **=    Assignment operators
    is is not    Identity operators
    in not in    Membership operators
    not or and    Logical operators