[jdev] High Volume & Temporary Files

[Dave Jarvis]

Hi,

I'm currently using Jabberd Version 1.4.2.  I used the quick setup on 
RedHat Linux, and am able to send messages between users easily and swiftly.

The reason I'm writing is because we're doing high volume (20,000) 
messages that are software driven into the Jabber server, with offline 
accounts.  This means that the XML spool files are growing quite large, 
even though the content of each message is quite small:

-rw-------    1 jabber   jabber       1.3M Mar 23 15:24 testing.xml

It takes 3 seconds for the jabber server to write a single message to 
the end of this file.  I've also noticed temporary files being written 
during this process, and (though I haven't looked at the source) believe 
the algorithm upon receipt of each new message is:

1) Copy all XML elements in the testing.xml file into the temporary 
file, up until the closing elements.
2) Append the new message onto the end of the temporary file.
3) Write the closing elements.
4) Rename the temporary file over the existing file.

When a user only has a few hundred messages, this algorithm works just 
fine, as the file sizes are small.  But when the files are a few MB in 
size (e.g., 7000+ messages), this algorithm suffers a severe performance 
penalty (between 3 and 5 seconds to add a message).  We're using a 
dual-CPU Intel-based machine running at 1.8 GHz per chip, with 4 chip 
emulation enabled, and a Linux bogomip count of 3578.26.  I don't know 
the stats on the harddrive.

If the file can be locked, I would recommend changing the algorithm:

1) Read from the end of the file until the last message
2) Truncate the file from the end of that message
3) Write the new message
4) Write whatever was truncated

Of course, you would want to keep a small journal for recovery purposes 
in case of a crash.

If there is a different version, or a different implementation of a 
Jabber server (which stores messages offline) which doesn't suffer from 
this performance issue, I'd really appreciate knowing.

Many thanks!

Sincerely,
Dave Jarvis
-- 
Invoq Systems Inc.
t. 250.380.0304
e. djarvis at invoqsystems.com
w. http://www.invoqsystems.com