[jdev] Re: Presence packets bottleneck on huge rosters

[Bresler, Jonathan]

Stephen, 

I don’t mean to antagonize, please forgive if I have gotten under your skin. Its hard to tell via email ;(  Just thinking about these issues in terms of scalability.  I have use IDX-Tsunami (http://tsunami.idealx.org/index.en.html) to put 7000 users on a  Compaq 1850R with 1GB RAM running jabberd-1.4.3.1 and jadc2s-0.9.0.1.  Stopped the test at that point as it was 3x my expected user population.  As an aside the same box (dual 550MHz P3 processors) does about 160 SSL'ed messages per second of about 85 bytes each.

You are correct that several <presence> messages (about 10) would be contained in each Ethernet packet amortizing the Ethernet, TCP and IP costs over that number of packets.  Perhaps my number were too round, too much of an overestimate. Still each stanza will be a separate write() to the socket via mio and from there to pth (have yet to read all the mio or any of the pth code).

A real-world application is a client at the end of a dialup link with a company wide roster.  I don’t have a solution to this.  I have added it to a list of things to "be aware of"  and "wonder/think about" when the chance arises.  

I'll take a look at stream compression....I still have a lot to learn here, so please excuse my newness and partial ignorance.  Stream compression would be very valuable for this as much of the <presence> stanzas are repeated data.  Compression should be rather high.

Jonathan

-----Original Message-----
From: jdev-bounces at jabber.org [mailto:jdev-bounces at jabber.org] On Behalf Of Stephen Marquard
Sent: Thursday, December 09, 2004 9:34 AM
To: jdev at jabber.org
Subject: [jdev] Re: Presence packets bottleneck on huge rosters


Bresler, Jonathan wrote:
> One practical consequence for the XML stream is a separate <presence>
> Stanza is sent from the server to the client for each contact.  I have
> To check the karma code to see if the client connection's karma is charged
> For this traffic.  
> 
> 500 contacts means 500 <presence> stanzas, each one being something like 
> 150 bytes longs (payload, not including Ethernet, TCP and IP headers, 
> call it 200 bytes in round numbers).  So we get 5 contacts per 1kB on the 
> Wire or 100kB, which is 800kb on the wire.  Rather heavy duty. ;(  But there
> Does not appear to be any choice at this time....so dribbling the presence
> Data out to the client over some time period appears to be desireable.

Although 1 stanza is not necessarily 1 ethernet packet, and 800KB is not 
much on a LAN for example.

Do you have a real-life application in mind where such large rosters 
would be used with frequent enough login/logouts to be an issue?

Another possibility for addressing this case efficiently is stream 
compression - http://www.jabber.org/jeps/jep-0138.html

Regards
Stephen

_______________________________________________
jdev mailing list
jdev at jabber.org
http://mail.jabber.org/mailman/listinfo/jdev

---
Incoming mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.804 / Virus Database: 546 - Release Date: 11/30/2004
 

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.804 / Virus Database: 546 - Release Date: 11/30/2004