Overview

We needed a message format for FlowBasedProgramming in ISconf, and an object class to support it.

See http://stein.cshl.org/software/boulder/ for an example data format for messages -- I've used this one for UNIX shell pipes many times. The problem with Boulder is that everything, both headers and payload, needs to be url-encoded. This can get quite slow with multi-megabyte payloads -- checking a 30 Mb Linux kernel tarball into ISconf would take minutes if we had to encode the whole thing.

What we're using instead is fbp822, developed specifically for ISconf but usable for anything else.

fbp822 Features

• Uses ASCII headers, and correctly handles conversion of boolean and integer data types.
• Supports a binary payload without encoding.
• Serializable in a language-portable way -- should be easy to write parsers in Perl, for example.
• Easy to nest one or more messages inside payload to get multi-part.

Example messages

The message format is pretty self-explanatory; if you stare at it long enough, you'll probably have correctly figured out how it works.

The following messages are copied directly from an ISconf 4.2.6 journal file. The first message records an isconf snap /tmp/2 command. Note the integer data types on the st_* parameters. This message has no payload; note the _size of 0. (Earlier in the ISconf data flow, the message carried the file content itself as the payload, but the payload gets stripped off and stored in block storage by the time the fbp message reaches the journal.)

From fbp822 Sun, 28 Aug 2005 06:21:58 -0000
_type: snap
st_mtime: 1125014381
_type_st_mtime: i
pathname: /tmp/2
st_mode: 33261
_type_st_mode: i
st_atime: 1125056671
_type_st_atime: i
st_uid: 0
_type_st_uid: i
st_gid: 0
_type_st_gid: i
xid: 1125210118.926761.0.294661@isconf11
message: root@isconf11: test (lock time Fri Aug 26 19:57:39 2005)
time: 1125210118
_type_time: i
blk: 59385ec6fddedb528d9ec4c860619c4c63516b7f-718af129f86bfdd2273ee077bf451239-1
_size: 0

This second example is the record of the command isconf exec /tmp/2 'hello "big" world!' . Note that the safest place to store the command's argv is in the message payload; this lets us easily delimit arguments with newlines while preserving embedded quotes.

From fbp822 Sun, 28 Aug 2005 06:29:24 -0000
_type: exec
_size: 26
message: root@isconf11: test (lock time Fri Aug 26 19:57:39 2005)
cwd: /tmp/roothome
xid: 1125210564.882450.0.124842@isconf11
message: root@isconf11: test (lock time Fri Aug 26 19:57:39 2005)
time: 1125210564
_type_time: i

/tmp/2
hello "big" world!