A normal thing to do is piping stdout to another command and sending stderr to the /dev/null bit bucket

command1 2>/dev/null | command2

But here is how to pipe stderr on and send stdout to /dev/null

command1 2>1 1>dev/null | command2

Using a named pipe we can redirect stdout in one way and stderr in another direction.

TMPD=$(mktemp -d)
mkfifo $TMPD/pipe
(command1 2>$TMPD/pipe | cat - > file.out) | command2 < $TMPD/pipe

To redirect all output inside a script

exec >file.log  2>1 # redirect stdout and stderr to the logfile

Another cute little trick is

VAR=$(< varfile)

instead of

VAR=$(cat varfile)

To watch a log file from the command line you’d normally use tail, you can do the same thing inside a script. (But be aware that tail lives on in a subshell and will need to be killed separately later.)

while read LINE; do
    if echo $LINE|grep -i "exiting" >dev/null; then
        echo "another daemon dies: $LINE"
    fi
done < <(tail -n0 -F /var/log/messages)

With the one command readarray we can read in a file and assign each line as elements of an array variable.

readarray FOO < foo.txt
echo line one is ${FOO[0]}
echo line two is ${FOO[1]}

Globbing of whole strings

mkdir -p /tmp/{cats,dogs}

sed doesn’t get enough appreciation!

sed -i "/from here/,/to there/ {
    s%^\(FirstName\ *\).*%\1$david\n%
    s%^\(Website\ *\).*%\1linuxsoftware.co.nz\n%
}" somefile.txt

How would you delete the last token in a file? Kind of tricky, but if we use tac to reverse the file we can change the problem to deleting the first token, then use tac again to reverse it back.

tac some.file|sed '0,/TOKEN/d'|tac > changed.file

And to add a watchdog to your script

(
 sleep 600            # if this script is still running in 10 minutes time
 kill -s SIGALRM $$   # send it an alarm signal
)&

OK, I think I have covered all the BASH goodies I wanted to now.

Heres some links to other resources which are useful.
• bash String Manipulations
• EnvironmentVariables
• Bash Parameter Expansion
• Bash Arrays
• Writing Robust Shell Scripts

For the last month or so I’ve been coding in BASH scripts. Of course I’ve used BASH before, but I’ve learnt a few things I just hadn’t known of before. So here are some of the new-to-me things I’ve come across.

local variables limited in scope to the inside of a function

function foo()
{
   local VAR="hi"
}

now VAR has scope only within foo.

Beware though local can only be used on a variable in a function so be careful about copying code from functions out into the main body of a script.

A variable expansion to convert a string to lower case

VAR=miXedCase
echo ${VAR,,}

or upper case

echo ${VAR^^}

Which is useful for doing a case-insensitive comparison

if [ "${V1,,}" == "${V2,,}" ]; 

Another useful shell variable expansion trick allows replacing part of the text in the expansion.

echo ${VAR//Case/Beans}

Another thing I’ve finally grokked is when to use [ ] in an if.

if executes a command (actually it could be a group of commands, more about that later) and if it returns 0 then runs the commands in the then part of the statement.

So, if you just want to check that a command ran ok, you can simply go.

if command; then
  echo ok
fi

which is the same as the more verbose

command
if [ $? -eq 0 ]; then
  echo ok
fi

[ is just another command. One that expects an expression followed by a ]. Look for it in /usr/bin. You could replace it with the test command which is the same thing.

i.e.

if [ "thing" != "another" ]; then
   echo "different things"
fi

is the same as

if test "thing" != "another"; then
   echo "different things"
fi

Having said that [ aka test is "just another command" I should admit that BASH actually has a copy "built-in" and so doesn't bother using the version in /usr/bin. So it is somewhat special, but the built-in and external versions are meant to be interchangable. man builtins for a list of all the commands which have built-in versions in BASH to speed execution up. There's a few.

What I've found a bit confusing is an expression can be built up either in one [ ] command, or as a group of [ ] commands.

i.e.

if [ "thing" != "another" -o $((1+1)) -eq 2 ]; then
   echo "something true"
fi

is the same as

if [ "thing" != "another" ] || [ $((1+1)) -eq 2 ]; then
   echo "something true"
fi

Notice the difference?

Inside the test command (in the [ ]) -o acts as an or, but in the shell || can be used to or different test commands together.

Interactive Shells vs non-interactive : Signals, process groups and such.

BASH programming is just doing stuff at the command line and then when it works pasting it into a file and calling it a program. But but but… When you are “doing stuff at the command line” the shell is in interactive mode, and some of the behaviour is different to a script running in a non-interactive shell.

One thing I have hit: in the interactive shell, when a command finishes it signals all of its children letting them know it is time to shut down now too.

We can manually do the same thing the interactive shell is doing with three commands

    trap "" SIGHUP       # ignore the signal ourselves
    kill -s SIGCONT 0    # wake up any sleepers
    kill -s SIGHUP 0     # the game is all over

I’ve normally used kill just on one process at a time (kill 5234 or kill %1). Or I’ve used killall to signal all process by program name (killall tail). But what the command kill -s SIGHUP 0 is doing is sending the HUP (hangup) signal to all processes in the process group.

What’s a process group David? It is a family of processes. By default each child process inherits the process group of its parent and thus we can deal with them as a group.

We start a new process group by starting a new process with the setsid command. This makes the new process the new leader of a a new process group.
kill 0 signals all the processes in our process group.
kill -- -leader_pid, signals all the processes in the process group led by the pid given (but note it is negative)

The double dash (--) is how we pass a dash to a BASH program without it being interpreted as a flag.

finally kill -- -1 is how every process in the whole system can be signalled (every process but the init process)
I guess we can think of it as one big process group that every process belongs to, led by the init process.

Here’s how to kill a command. We nicely just send the TERM signal at first giving it time to tell its children and clean itself up. But if it ignores that, then we kill it and and its family with the KILL signal which it cannot ignore.

    COMMAND=$(ps -p$PID -o comm=)
    if [ -n "$COMMAND" ]; then
        echo "Killing old $COMMAND command pid=$PID "
        kill -s SIGTERM $PID
        sleep 1
        if ps -p$PID >/dev/null; then
            # give it a few seconds more
            sleep 10
            if ps -p$PID >/dev/null; then
                # kill the whole process group without mercy
                kill -s SIGKILL -- -$PID
                kill -s SIGKILL $PID
            fi
        fi
    fi

Fun with mailx
This example shows how to send mail via a particular smtp server.

echo -e "$MSG\n"|cat - $LOG|from=sender@example.com smtp=mailserver.example.com mailx -s "$SUBJ" "$TO"

Playing with tput
The tput command allows us to do funky things on the screen in a reasonably terminal-independant way.

tput -S <<EOF
clear
cup 2 10
bold
EOF
echo "Look at me!!!"
tput cup 5 0
tput sgr0

I’m going to stop here and start again another day in a new post. Coming up: redirecting streams beyond just sending them to /dev/null, loading a file with readarray, and watching log files.

The Auckland Linux Users’ Group meets monthly on a Monday night (normally 7.30pm on the first Monday of the month). The last meeting was Monday 9th June with Martin Kealey talking on advanced BASH.

Dropped in at the library after work and had a beer or three with Simon at the London Bar before heading up. At the old Delphi Users’ Group, Borland used to ply us with beer and pizza, but the LUG is a BYO biscuits and coke event – maybe IBM or Novell would like to sponsor refreshments?

Turned out our room in AuckUni’s new Business School was double booked. But Colin had a room available in Engineering, and then when Martin couldn’t get through the firewall to his server Vince volunteered his laptop to demonstrate BASH. Shows the Linux community spirit.

A few things I noted down as new to me were:

• Bash arrays: A = (1 2 3 4); echo ${A[0]}
• xargs with find is more efficient (so faster) than using -exec
• keyboard bindings
• double square braces [[ are built in operators which can be used instead of the test [ command, e.g. if [[ built && in ]]

The top 10 commands from your shell meme. Here’s mine (from my server rata):

history | awk '{print $2}' | awk 'BEGIN {FS="|"} {print $1}' | sort | uniq -c | sort -nr | head -10
159 ls
117 cd
59 scp
29 gvim
28 ssh
24 mv
21 mysql
20 locate
19 su
13 rm

So what does this say?

• ls most of the time I’m only looking
• cd then I’m gone
• scp the data is always greener on another host
• gvim I do do some work
• ssh but I wish I was somewhere else
• mv maybe moving things around will help?
• mysql ok, just lately I’m playing around with MySQL replication, more about that later
• locate I’m so lost
• su time to call upon a higher power
• rm when all else fails try something destructive