[JDEV] Stress testing and connection limits

[Eric Bowersox]

> I've been stress testing Jabber to get a feel for how many 
> concurrent users
> it can handle. I've hit some limit on the number of 
> connections that Jabber
> will accept. Hopefully, it's a soft limit in Linux or Jabber, 
> or maybe a
> mistake in my test.
[...]
> This works great up to about 1000 (1024?) clients
> (threads). Above that number, I start seeing all kinds of 
> failures. I think
> it may be caused by pth_accept rejecting the connection, but 
> I'm not sure.
> Anyone have any ideas? I did disable the connection rate 
> check in tlisten.c.
> Hopefully, it's something like a 1024 limit on the number of 
> connections
> from the same IP.

You've hit the nail right on the head.  It's not pth_accept() that's the
problem, though, it's pth_select(), which relies on the select() system call
in the OS.  Many (most?) implementations of select() won't handle more than
1024 file handles.

The obvious solution is to use poll() instead, but the select() is buried
down inside the pth library, which falls firmly into the realm of Code We
Don't Dare Tamper With Because It's Impossible For Mere Mortals To
Understand.

We (the jserver developers, that is) are currently developing a solution
that uses additional processes as "multiplexers" to aggregate large amounts
of connections.  With this, it would be  theoretically possible to increase
the number of server connections to as many network sockets as your system
will handle (which is more like 64K).  This isn't ready yet, though; if
someone knows different, please step in and correct me...

					Eric

--
Eric J. Bowersox - Jabber Inc. - ebowersox at jabber.com
<mailto:ebowersox at jabber.com>   http://www.jabber.com
OpenProjects IRC #jabber: erbo  - Advogato: Erbo
"AIM is to Jabber as Notepad is to emacs" - washort, #jabber