[jdev] Financial messaging via XMPP

[Martin Sustrik]

Dave,

Thanks for extensive clarification.

I believe I have some idea of how XMPP plugin can be implemented now. 
When we have something working I'll post the performance figures on this 
list.

The issues that still may be performance bottlenecks are:

1. Size of XMPP wrapper of the binary message - 360 bytes in the example 
in the article - with financial protocol like FIX/FAST the binary data 
tend to be quite small (30-40 bytes) thus 360 bytes of wrapping XML can 
extend the message size tenfold.

2. Compression and decompression (binary->base64->TLS compression->TLS 
decompression->base64->binary data) may cause latency to be worse. 
However, I am not sure what exactly the impact would be.

Martin

Dave Cridland wrote:
> I was going to wade in on this one sooner, but I wanted to read the 
> AMQP specification first. It strikes me that the bulk of the 
> specification actually maps onto XML (and, by inference, XMPP) quite 
> well, so I'm a little surprised by the conclusion you draw.
>
> I apologise for answering points that are already answered by other 
> people - I've seen Maciek's reply but left my similar comments in - 
> and I apologise if I've got bits of AMQP wrong - I've spent not much 
> more than an hour learning it.
>
> On Fri Jan 25 15:02:32 2008, Martin Sustrik wrote:
>> 1. XMPP can be used for sending opaque messages, however, there are 
>> several limitations:
>> 2. Binary content has to be translated into Base64, adding 1/3 of 
>> overhead to message size
>
> Not really - assuming the binary message is incompressible to begin 
> with, the base64 encoding is recovered through compression - XMPP 
> gives you this both by XEP-0138 and by TLS. On the assumption that you 
> care about bandwidth, you'll have compression on, and your messages 
> will be compressed anyway. Equally, they'll potentially be end-to-end 
> encrypted, too.
>
> Rather oddly, AMQP contains nothing about either compression nor 
> encryption. I'd have thought the latter would be important in 
> financials. (Oh, and that SASL profile is incomplete - SASL can 
> negotiate encryption too, but there's no indication of where this 
> might kick in, nor any MTI mechanisms).
>
>
>> 3. There's no way to do zero-copy as the message has to be translated 
>> on both ends of the connection
>
> Sure. But then again, a lot of the data in AMQP needs to be copied 
> anyway to circumvent alignment issues, handle buffering, queuing, etc. 
> Unless you're sending really big chunks of longstr data about, the 
> impact is likely to be low - and even if you are, it's just one more 
> codec.
>
> For servers, where this would be the highest impact, it seems very 
> unlikely that any data is going to be examined anyway - servers look 
> at routing keys and topics, and more or less leave everything else 
> untouched, so it'd seem reasonable to stick topics and routing keys in 
> XML, and then leave the message payload as a blob.
>
> All this assumes that the messages can't be formatted as XML, of course.
>
>
>> 4. All the messages have to be acknowledged. No no-acknowledge mode 
>> exists.
>
> <message/> is unacknowledged, <iq/> is acknowledged.
>
> (To put it another way, <iq/> are like your Request/Response frames.)
>
>
>> 5. There's no batch acknowledge functionality (acknowledging sequence 
>> of messages using single acknowledge).
>
> Depends what you mean by this. If you use <iq/>, then no, but then 
> again, these can be acknowledged end-to-end out of order, just as 
> Request/Response frames do in AMQP.
>
> If you mean across a connection, then look at XEP-0198, which provides 
> precisely this, as well as several other useful features for 
> reliability. (And much better than AMQP's heartbeat frames, too, IMO).
>
>
>> 6. Pipelining of acknowledges is allowed, but discouraged. ("The 
>> sender need not wait for these acknowledgements before sending 
>> further stanzas. However, it is RECOMMENDED that the sender does wait 
>> in order to minimize possible rate-limiting penalties.")
>>
>>
> That's a RECOMMENDED - a SHOULD not a MUST - and only for bytestreams. 
> If you're sending <message/> stanzas with binary content, there's no 
> need to concern yourself with whether the endpoint is acknowledging, 
> since it won't be. Worth noting that bytestreams are for arbitrary 
> length data - while AMQP does in principle allow frames of several G 
> (assuming they're made up of multiple longstrs, for example), I'm 
> under the impression that individual frames tend to be relatively 
> small, and would map to a single XMPP stanza.
>
> I'd suggest sitting down and examining RFC3920bis, and familiarizing 
> yourself with the general concepts. XMPP will not be as efficient as 
> AMQP, but the difference need not be huge. On the other hand, the 
> potential gains coming from using XMPP are very great indeed, giving 
> you improvements in security and multiple sites to name but two.
>
> I hope this helps.
>
> Dave.