LinuxSoftware

Coding and tramping in Aotearoa / New Zealand

Car insurance

May 16

Stream Formats

, , , david, Monday, 9:40 pm

A technique to support multiple formats in C++ Stream I/O.

I was writing a class for a kind of status message which could be output in either XML or a plain text format. I wanted the class to be usable with C++ stream operators << and >>, but because there were two formats it wasn’t easy to see which one the stream operators should work with.

I remember when I first encountered C++ Streams I was impressed by the elegance of the design. The Borland C++ Programmers Reference manual had a great explanation of the concept. [Unfortunately I'm not sure what happened to my copy, so I'll just give my own crappy description here.] A stream is built up from a source or a sink, manipulators and the ‘objects’ which are being streamed, all connected with the put-to (<<) or get-from (>>) operators. I still consider it superior to the clumsy flags and varargs approach of scanf and printf. But despite this it is not often that I define put-to and get-from operators for my own classes. Instead I’ve often ended up defining custom input and output functions. Hopefully the technique below will make it easy for me to use stream operators instead.

template <typename T, typename U>
struct StreamFormat
{
    StreamFormat(T& t) : m_t(t) {}
    T& m_t;
};

StreamFormat is a template which just holds a reference to some object of type T, but it is also disambiguated by the format type U.

What this means is for a type we can define multiple format type wrappers.

   typedef StreamFormat<Thing, struct Xml> AsXml;
   typedef StreamFormat<Thing, struct String> AsString;

The struct Xml and struct String are incomplete types which are just used to differentiate the format types. (They’re never used again once they’ve served that purpose.)

Now we have an interface which which can support different formats of stream I/O.

    thingss >> Thing::AsString(thing);
    cout << Thing::AsXml(thing) << endl;

But how is this I/O implemented? The StringFormat template is accompanied by generic stream operators

template <typename T, typename U>
istream& operator >>(istream& lhs, StreamFormat<T,U> rhs)
{
    return rhs.m_t.getFrom(lhs, rhs);
}
template <typename T, typename U>
ostream& operator <<(ostream& lhs, StreamFormat<T,U> rhs)
{
    return rhs.m_t.putTo(lhs, rhs);
}

And as shown these delegate back to the wrapped type’s getFrom and putTo member functions which will be overloaded by a StreamFormat argument. These do the real work, and as they are member functions have direct access to the member variables.

istream& Thing::getFrom(istream& in, AsXml)
{
    // world's dirtiest XML parsing
    in.ignore(numeric_limits<streamsize>::max(), '>');
    in >> m_a;
    in.ignore(numeric_limits<streamsize>::max(), '>');
    return in;
}
ostream& Thing::putTo(ostream& out, AsXml) const
{
    out << "<thing>" << m_a << "</thing>";
    return out;
}
istream& Thing::getFrom(istream& in, AsString)
{
    in >> m_a;
    return in;
}

You might be asking “why not just call the getFrom and putTo methods directly?”. To which I say “didn’t you read what I said about the elegance of the stream operators?!” and “having them as stream operators makes them easy to mix with other standard C++ objects” and “they can be used in standard algorithms with the istream_iterator and ostream_iterator wrappers”.

I did think about having member functions which would construct the StreamFormat objects.

AsXml asXml() { return AsXml(*this); }

They make things a little bit neater, but I don’t think they’re really worth it.

thingss >> thing.asXml();

Finally here’s all the code of a little test program I wrote. Does anyone else use this technique? Is there a better way? Or do you think it is all unneeded syntactic sugar? “mmmmm sweet sweet syntactic sugar.”

#include <iostream>
#include <sstream>
#include <limits>

using namespace std;

template <typename T, typename U>
struct StreamFormat
{
    StreamFormat(T& t) : m_t(t) {}
    T& m_t;
};
template <typename T, typename U>
istream& operator >>(istream& lhs, StreamFormat<T,U> rhs)
{
    return rhs.m_t.getFrom(lhs, rhs);
}
template <typename T, typename U>
ostream& operator <<(ostream& lhs, StreamFormat<T,U> rhs)
{
    return rhs.m_t.putTo(lhs, rhs);
}

class Thing
{
 public:
   Thing() : m_a(0) {}
   Thing(int a) : m_a(a) {}

   int geta() const { return m_a; }

   typedef StreamFormat<Thing, struct Xml> AsXml;
   istream& getFrom(istream& in, AsXml);
   ostream& putTo(ostream& out, AsXml) const;

   typedef StreamFormat<Thing, struct String> AsString;
   istream& getFrom(istream& in, AsString);

 private:
   int m_a;
};

istream& Thing::getFrom(istream& in, AsXml)
{
    // world's dirtiest XML parsing
    in.ignore(numeric_limits<streamsize>::max(), '>');
    in >> m_a;
    in.ignore(numeric_limits<streamsize>::max(), '>');
    return in;
}
ostream& Thing::putTo(ostream& out, AsXml) const
{
    out << "<thing>" << m_a << "</thing>";
    return out;
}
istream& Thing::getFrom(istream& in, AsString)
{
    in >> m_a;
    return in;
}

int main(int argc, char *argv[])
{
    Thing thing;
    stringstream thingss;

    thingss.str("<thing>99</thing>");
    thingss >> Thing::AsXml(thing);
    cout << thing.geta() << endl;

    thingss.str("86");
    thingss >> Thing::AsString(thing);
    cout << Thing::AsXml(thing) << endl;

    return 0;
}

Apr 30

Advanced BASH programming #2

, , , david, Saturday, 4:20 pm

Shell.

A normal thing to do is piping stdout to another command and sending stderr to the /dev/null bit bucket

command1 2>/dev/null | command2

But here is how to pipe stderr on and send stdout to /dev/null

command1 2>1 1>dev/null | command2

Using a named pipe we can redirect stdout in one way and stderr in another direction.

TMPD=$(mktemp -d)
mkfifo $TMPD/pipe
(command1 2>$TMPD/pipe | cat - > file.out) | command2 < $TMPD/pipe

To redirect all output inside a script

exec >file.log  2>1 # redirect stdout and stderr to the logfile

Another cute little trick is

VAR=$(< varfile)

instead of

VAR=$(cat varfile)

To watch a log file from the command line you’d normally use tail, you can do the same thing inside a script. (But be aware that tail lives on in a subshell and will need to be killed separately later.)

while read LINE; do
    if echo $LINE|grep -i "exiting" >dev/null; then
        echo "another daemon dies: $LINE"
    fi
done < <(tail -n0 -F /var/log/messages)

With the one command readarray we can read in a file and assign each line as elements of an array variable.

readarray FOO < foo.txt
echo line one is ${FOO[0]}
echo line two is ${FOO[1]}

Globbing of whole strings

mkdir -p /tmp/{cats,dogs}

sed doesn’t get enough appreciation!

sed -i "/from here/,/to there/ {
    s%^\(FirstName\ *\).*%\1$david\n%
    s%^\(Website\ *\).*%\1linuxsoftware.co.nz\n%
}" somefile.txt

How would you delete the last token in a file? Kind of tricky, but if we use tac to reverse the file we can change the problem to deleting the first token, then use tac again to reverse it back.

tac some.file|sed '0,/TOKEN/d'|tac > changed.file

And to add a watchdog to your script

(
 sleep 600            # if this script is still running in 10 minutes time
 kill -s SIGALRM $$   # send it an alarm signal
)&

OK, I think I have covered all the BASH goodies I wanted to now.


Apr 27

Advanced BASH programming #1

, , david, Wednesday, 9:30 pm

Shell.

For the last month or so I’ve been coding in BASH scripts. Of course I’ve used BASH before, but I’ve learnt a few things I just hadn’t known of before. So here are some of the new-to-me things I’ve come across.

local variables limited in scope to the inside of a function

function foo()
{
   local VAR="hi"
}

now VAR has scope only within foo.

Beware though local can only be used on a variable in a function so be careful about copying code from functions out into the main body of a script.

A variable expansion to convert a string to lower case

VAR=miXedCase
echo ${VAR,,}

or upper case

echo ${VAR^^}

Which is useful for doing a case-insensitive comparison

if [ "${V1,,}" == "${V2,,}" ]; 

Another useful shell variable expansion trick allows replacing part of the text in the expansion.

echo ${VAR//Case/Beans}

Another thing I’ve finally grokked is when to use [ ] in an if.

if executes a command (actually it could be a group of commands, more about that later) and if it returns 0 then runs the commands in the then part of the statement.

So, if you just want to check that a command ran ok, you can simply go.

if command; then
  echo ok
fi

which is the same as the more verbose

command
if [ $? -eq 0 ]; then
  echo ok
fi

[ is just another command. One that expects an expression followed by a ]. Look for it in /usr/bin. You could replace it with the test command which is the same thing.

i.e.

if [ "thing" != "another" ]; then
   echo "different things"
fi

is the same as

if test "thing" != "another"; then
   echo "different things"
fi

Having said that [ aka test is "just another command" I should admit that BASH actually has a copy "built-in" and so doesn't bother using the version in /usr/bin. So it is somewhat special, but the built-in and external versions are meant to be interchangable. man builtins for a list of all the commands which have built-in versions in BASH to speed execution up. There's a few.

What I've found a bit confusing is an expression can be built up either in one [ ] command, or as a group of [ ] commands.

i.e.

if [ "thing" != "another" -o $((1+1)) -eq 2 ]; then
   echo "something true"
fi

is the same as

if [ "thing" != "another" ] || [ $((1+1)) -eq 2 ]; then
   echo "something true"
fi

Notice the difference?

Inside the test command (in the [ ]) -o acts as an or, but in the shell || can be used to or different test commands together.

Interactive Shells vs non-interactive : Signals, process groups and such.

BASH programming is just doing stuff at the command line and then when it works pasting it into a file and calling it a program. But but but… When you are “doing stuff at the command line” the shell is in interactive mode, and some of the behaviour is different to a script running in a non-interactive shell.

One thing I have hit: in the interactive shell, when a command finishes it signals all of its children letting them know it is time to shut down now too.

We can manually do the same thing the interactive shell is doing with three commands

    trap "" SIGHUP       # ignore the signal ourselves
    kill -s SIGCONT 0    # wake up any sleepers
    kill -s SIGHUP 0     # the game is all over

I’ve normally used kill just on one process at a time (kill 5234 or kill %1). Or I’ve used killall to signal all process by program name (killall tail). But what the command kill -s SIGHUP 0 is doing is sending the HUP (hangup) signal to all processes in the process group.

What’s a process group David? It is a family of processes. By default each child process inherits the process group of its parent and thus we can deal with them as a group.

We start a new process group by starting a new process with the setsid command. This makes the new process the new leader of a a new process group.
kill 0 signals all the processes in our process group.
kill -- -leader_pid, signals all the processes in the process group led by the pid given (but note it is negative)

The double dash (--) is how we pass a dash to a BASH program without it being interpreted as a flag.

finally kill -- -1 is how every process in the whole system can be signalled (every process but the init process)
I guess we can think of it as one big process group that every process belongs to, led by the init process.

Here’s how to kill a command. We nicely just send the TERM signal at first giving it time to tell its children and clean itself up. But if it ignores that, then we kill it and and its family with the KILL signal which it cannot ignore.

    COMMAND=$(ps -p$PID -o comm=)
    if [ -n "$COMMAND" ]; then
        echo "Killing old $COMMAND command pid=$PID "
        kill -s SIGTERM $PID
        sleep 1
        if ps -p$PID >/dev/null; then
            # give it a few seconds more
            sleep 10
            if ps -p$PID >/dev/null; then
                # kill the whole process group without mercy
                kill -s SIGKILL -- -$PID
                kill -s SIGKILL $PID
            fi
        fi
    fi

Fun with mailx
This example shows how to send mail via a particular smtp server.

echo -e "$MSG\n"|cat - $LOG|from=sender@example.com smtp=mailserver.example.com mailx -s "$SUBJ" "$TO"

Playing with tput
The tput command allows us to do funky things on the screen in a reasonably terminal-independant way.

tput -S <<EOF
clear
cup 2 10
bold
EOF
echo "Look at me!!!"
tput cup 5 0
tput sgr0

I’m going to stop here and start again another day in a new post. Coming up: redirecting streams beyond just sending them to /dev/null, loading a file with readarray, and watching log files.


Apr 4

Updated my Vim plugins

, , david, Sunday, 2:52 pm


Vim is a powerful text editor, based on the standard Unix vi command, but much improved. The different modes are confusing to new users, but when you’ve been using them as long as I have they just seem natural. My fingers have learned vi and don’t like having to use anything else. I end up with strings like jjjbcw in other documents.

I’ve built up a bundle of plugins and configurations which I use with Vim. Recently I’ve gone through, cleaned out the ones I never use, updated others and added new ones. If you’re interested you can download the bundle or check out my Vim wiki page for more details.

To install the bundle: untar it, then rename dot.vim to .vim, dot.vimrc to .vimrc, and dot.gvimrc to .gvimrc, being careful to first rename your existing vim files if you have anything you want to keep. There’s not much in my .vimrc and .gvimrc, so you don’t even really need to bother with them. Most of my configuration settings are in my vls-config.vim and vls-keys.vim plugins.

Cool new plugins included are: PyFlakes for on-the-fly Python syntax checking; TagList, a function/source code browser, most recently used (MRU) files, Case-persistant substitutions, and the NERDTree file explorer. I’ve also created myself acustom toolbar which reminds me to use some of the new features of Vim 7.0 such as folding, tabs, and the undo tree. Happy Viming!


Jun 12

Running Fedora 11

, , , , , , david, Friday, 8:10 pm


I’ve been running Fedora 11 at home and work for a three days now, and I am very impressed. With Linux being a very mature product and Fedora releasing twice a year, an upgrade isn’t the radical experience it was nine or ten years ago, when Linux was raw and new. Nowadays each release is an incremental improvement.

There are new versions of the familiar apps: Open Office 3.1.0, Firefox 3.5, Gnome 2.26.1, GCC 4.4.0, Python 2.6, Gimp 2.6, Pidgin 2.5.5. Open Office can open those annoying .docx files. Gnome’s menus seem a bit tidier. PulseAudio is meant to be much better. Will hopefully solve my friends’ sound problems, but it was alright on my laptop before, so I haven’t noticed that difference.

Boot times are faster. The goal for Fedora 11 was to be at the login screen in 20 seconds. On my laptop it takes 30 seconds, but that’s still pretty good. My laptop is a Dell D620 with, I’m very happy to report, suspend and resume working perfectly on it. Also fixed on my laptop is an annoying problem Fedora 9 had with simulated middle mouse button clicks. Now either the top two or the bottom two touchpad buttons can be used. Power usage is meant to be optimised, haven’t tested if this means I get an extended battery life yet.

The kernel installed is 2.6.29.4. This is a bit of a problem at work, as VMware-Player can’t build modules to run against this version. Ubuntu, openSuse and Gentoo users have also hit this problem. There are long, complicated, discussion forum threads with different patches which claim to fix things, but don’t work for me. Eventually VMware will catch up and release a new version of VMware-Player which will work with the 2.6.29 kernel. It’s a pain to wait, but that’s what you get with kernel code which doesn’t live with the rest of code in the kernel source tree.

Fedora 11 has the new ext4 filesystem format as default. This should make file access faster. I was working on ext4 support at my previous job for their product. I’m a bit surprised the Fedora installer (Anaconda) won’t allow the root filesystem to be ext4. ext4 can support a mix of on-disk structures (pind/dind/tind blocks or extents) and I thought this allowed Grub to still boot from it. Oh well, ext3 will do for /boot until Grub2 is finished (or the world ends, whichever comes first).

Lastly the fonts look fantastic. I’m not sure exactly what has changed there. Fedora has had nice fonts for a while, but text is looking very smooth these days. Though I did have problems with DDD (the GDB debugger front end). The DDD user interface hasn’t changed much for many many years and obviously can’t use the new fonts. It was displaying very large ugly letters, so I installed a bunch of the old font packages like xorg-x11-fonts-100dpi. That’s fixed DDD so it is at least acceptable to work with.

I’m pretty pleased with how the upgrade has gone. All up Fedora 11 seems a bit faster, more polished and nice and stable. I recommend giving it a try.


Next Page »