[jdev] Properly filtering broadcast events

[Pedro Melo]

On Nov 26, 2008, at 6:52 PM, Bryan Morgan wrote:

> I'm new to XMPP but am very excited about the possibilities.  We are  
> designing an application based on XMPP and want to be sure we are  
> leveraging the appropriate technologies and standards in the correct  
> places, in order to ensure scalability as well as future-proof the  
> app, as much as is possible.  My question, at its most basic is what  
> is the recommended approach for handling the following scenario:
>
> (1) Users starts session
> (2) Users submits message (via HTTP or XMTP - doesn't really matter)
> (3) Server picks up message (via asynchronous MQ or via XMPP  
> component - doesn't really matter....I think)
> (4) Server needs to flexibly broadcast message out to a large number  
> of users but (and this is key) this list of users needs to be  
> dynamically filtered
>
> I originally was hoping PubSub would work but that seems to rely on  
> a fixed node with a fixed set of subscribers.  In my scenario, I may  
> have fixed nodes but I don't want to push a message out to all  
> subscribers - just a dynamically chosen subset of those (based on  
> some filter parameters).  Some options I've considered include:
>
> * Pushing out the messages to all members of the node and letting a  
> custom XMPP client filter them.  This makes the server simple but  
> results in possibly many messages being sent to many "receiving"  
> clients when a subset would have worked.  Considering there may also  
> be a large volume of user-submitted messages coming in to trigger  
> this, minimizing the number of outbound messages is critical from a  
> scalability standpoint.
>
> * Not sure if this is possible but in the routing chain could we  
> build a component that would intercept the pubsub messages and apply  
> the filtering there?  Not sure if it's possible to do this generally  
> for "all broadcast recipients" or if the code would have to be  
> called once for every message recipient which would then again kill  
> performance/scalability as the load grows.
>
> * The inbound item picked up off the queue could also be used to  
> determine the list of valid recipients, and then.....I  
> assume....loop through them one at a time and send them individiual  
> messages??  Again, seems to work at a small scale but maybe not at a  
> large scale?  Is there a way to build a dynamic list of users and  
> have XMPP send messages to that list?  Are there practical  
> limitations on handling the size of this list or are there  
> "sharding" approaches for big XMPP broadcasts like this?
>
> Any advice or lessons learned by XMPP veterans would be greatly  
> appreciated.  I apologize in advance for the lengthy message.

Of the top of my head I don't see any out-of-the-box solution for what  
you want.

I would build a pub-sub-like component (reuse the parts of XEP-0060  
that you need), and implement the broadcast myself. You should split  
the destination list in chunks and have a XMPP worker component deal  
with each chunk.

Try and make sure that S2S connections are ready before you start  
broadcasting (ping or disco#info the remote server before you send  
them messages). This should cut a bit on S2S buffering or even dropped  
messages (depends on server).

Make use of component load-balancing built in ejabberd, XCP. If your  
server does not support this, take a look at a recent load balancing  
proxy written in Python/Wokkel. There was a C version of the same  
concept but I failed to find the link right now.

Building large multicast systems over XMPP right now is complicated if  
you use federation. Large sites like GTalk, Jabber.org and others will  
end up receiving large bursts of messages from you and trigger rate- 
limits, so the end user experience might suffer. Assume that you'll  
have to provide local accounts for users who require QOS.

Apart from that, have the appropriate amount of fun.

Best regards,
-- 
Pedro Melo
Blog: http://www.simplicidade.org/notes/
XMPP ID: melo at simplicidade.org
Use XMPP!