[JDEV] performance numbers.. questions

[Dustin Puryear]

Oliver Jones wrote:
> If I understand you, the sequence of events is this, for each user:
> 
> a. connect
> b. log in.
> c. send x messages as fast as possible.
> d. stay connected, but stop sending messages.

Yes.

> When you run the test, you run these steps for each virtual user in
> turn.  Right?

Yes.

> Some questions about your test sequence:
> --Does the log-in step also create an account for the virtual user?

No. I built a tool named reg_users that creates users test_0..test_n-1.
I didn't want to build account creation code into each test module.
reg_users would be another good place to time jabberd. Specifically, how
fast can new users be created?

> --What kind of authentication are you using?

Plaintext. However, I could easily extend the test clients to use one of
several methods.

> --To what user do the x messages get sent?

Self. 

> --Is the recipient online or are the messages getting stashed in jspool?

Online. Again, I could easily extend the test client to NOT set presence
information, which would cause jabberd to store messages offline.

> --Does the volume of the x messages sent cross the karma threshold?

I have disabled (or tried to) karma in jabber.xml and rate-limiting in
jabberd/lib/rate.c. Naturally, you could alter jabber.xml and note
differences in the numbers returned (min-message-delivery,
max-message-delivery, average-message-delivery, and
total-message-delivery).

> You proposed this scenario:
> 
> >for user 1..n do
> >   connect user to jabber
> >   send x messages and track delivery times
> >done
> >
> >for user 1..n do
> >   disconnect user from jabber
> >done
> 
> I'd like to see tests that figure out the load that comes on when many
> users connect and authenticate at once.  The question to ask might be,

Easily accomplished. Simply run multiple instances of the test client
concurrently! You can then merge the client output, and naturally you
still only need a single set of server performance numbers. I have
considered creating a network heartbeat so that my clients can sync up
when running multiple instances concurrently, but I don't think that's
really required at this point. Plus, if you want to max out jabberd then
the network heartbeat would have to be *fast*, and that would introduce
network issues that may alter your results. 

By using the UNIX join tool you can just graft the output files together
and the graphs come out nice.

>   -- when 500 users are already connected to jabber but not doing much,
>      how many new user connections can jabber accept each second?

Good question.

> A subsidiary question might be how the choice of authentication method
> affects this, and how the choice of xdb_file or other jspool module affects it.

Another good question. The authentication method was addressed above. As
far as xdb_file and jspool, that is simply a matter of the client NOT
setting presence, and forcing jabberd to store offline messages. I
suppose I could create a  command-line flag to specify presence
information.

> Another subsidiary question might be how many NEW user accounts can be
> created in a second.

Refer to reg_users.

> I'd like to see tests that figure out the load that comes on when many
> connected users send lots of messages.  The question to ask might be,
> 
>    -- when 1000 users are already connected, how many messages a second
>        can each user send to some other user -- what's the message throughput?

Good test. To me this is partially addressed by active_jab already. The
real issue is how many concurrent users can send messages, and at what
rate? This can be tested using multiple instances of active_jab, but at
some point that may become a hassle. Ideas are welcome.

> >CPU=======
> >Big surprise here. Jabber seems to *quickly* begin eating CPU cycles as
> >the number of connected users increases. Here is some data:
> 
> Our jabberd and jpollds take very little CPU time, but our users stay
> connected for long periods and typically don't send huge numbers of
> messages -- dozens a day.
> 
> Your high CPU usage may be due to overhead on new connections, or overhead
> on karma-throttling virtual users who are sending big bursts of data.

I am pretty sure karma should NOT be affecting my test as I have <dec>
and <penalty> set to 0. I may be wrong though.

So, the following tests are in order:
1 logon times based on authentication method
2 message delivery times for 1 user
3 message delivery times for n users

Any other tests?

2 and 3 differ in that 2 will test how fast I can pump messages across
jabberd period. Test 3 will test how fast I can pump messages for n
concurrent users. This differs from active_jab since active_jab connects
and test user message delivery in a serial fashion. Test 3 may be a bit
more work.

Naturally, what test I create will depend on what numbers my work needs.
However, I absolutely welcome additions to my code and test modules.

Regards, Dustin

-- 
Dustin Puryear <dpuryear at usa.net>
http://members.telocity.com/~dpuryear
In the beginning the Universe was created. 
This has been widely regarded as a bad move. - Douglas Adams