[jdev] Announcing Gradient / Scaling between federated servers

[Ian Sollars]

Hi everyone,

I know there is some interest in moving SVG over XMPP for the purposes
of whiteboarding. I've just gone live with some code that does
something similar, that may be interesting to anyone looking at this.
The URL:

http://ex-337.net/gradient/

The code is built on the Smack API, in conjunction with Apache Batik,
Mozilla Rhino and the Jaxen XPath library.

In brief, SVG documents are loaded via XMPP (which gives rise to a
scaling problem, which I have a question about below), and once
loaded, the SVG document has access to a scripting environment that
can send and receive XMPP stanzas.

The document effectively can send messages and make RPC (IQ) calls to
the JID it was loaded from, and any other JIDs it has been subscribed
to. The server JID can also make RPC calls against the document, and
send message & presence stanzas to several documents at once. This is
all explained under "Targeting data and RPC" under the documentation.

As well as sending data that will be processed by client-side
JavaScript functions (if declared), the server JID can also send or
broadcast modifications to be made to client-side documents. Put
simply, one of the extensions allows a server JID to remotely apply
element-level modification operations equivalent to the four main DOM
methods - appendChild, insertBefore, replaceChild and removeChild - at
points within the document that are specified by XPath expressions.
This is explained in detail under "Docment updating".

This is a brief description of how the client and server interact.
There's more detail on the website for anyone who's interested, and
I''d be happy to listen to any comments or feedback.

As cool as this is, this is a little issue with scaling.

To be specific, there is a problem with the document request
architecture: the 'client' and 'server' are both equal in the eyes of
jabber.org, i.e. they are both normal client JIDs. This is going to
break down if 10,000 other 'client' JIDs all send GET IQs to (e.g.)
demo at example.org/gradient. There is an inherent scaling problem in
routing lots of stanzas through one TCP stream.

I don't yet have a concrete solution to this, but my first reaction
was to move the 'server' JID to within the XMPP server - i.e. the
sever would have a JID of more similar to a MUC room, such as
"demo.example.org/gradient". My thinking was to federate an
'application' XMPP server that would host these JIDS with an
'authentication' XMPP server to which users sign on, manage rosters
etc.

For example, suppose I set up ian.net with Jabberd2. Users can sign
on, chat with other users, etc. When Bob has an app that he wants to
make available to the general public, he federates bob.net with
ian.net, and thereafter (assuming I understand what federation is) GET
IQs sent from users on ian.net to a JID such as "app.jabber.bob.net"
are routed via the federation between the two servers on port 5269.

My question is, how do federating servers scale to handle large
amounts of traffic? Would the architecture scale any better than if he
had simply connected as app at jabber.ian.net?

I get the feeling I may be looking at this the wrong way. I'm
interested in listening to anyone who knows more about XMPP and has
ideas about how to approach this.

Regards,

 Ian