[JDEV] Sparse considerations about server status
Dustin Puryear
dpuryear at usa.net
Thu Aug 16 13:41:48 CDT 2001
I had some time yesterday and today to play with jabbertest and jabberd
1.4.1 and here are my results.
First, my changes to jabbertest: both the msgloadsnd and msgloadrec
clients are now checking for closed sockets. In addition, msgloadsnd is
watching for bounced messages.
Okay, so I first tested my socket handling code by running jabbertest
with 5 user pairs sending 3 messages/sec for 60 seconds. Once the test
begins I manually kill jabberd. Both the msgloadsnd and msgloadrec
clients report that their sockets were closed and then abort. So that
works.
I then restart jabberd and run jabbertest with 200 user pairs sending 2
messages/sec for 60 seconds, which gives me the following stats:
UserPs Rate Duration MinDelTime MaxDelTime AvgDelTime MsgCnt ExpMsg
MsgLossRate
200 2 60 0.01704 440.68959 151.83634 14322 24000
0.40325
That's a 40% loss rate for a 400 messages/sec test. I look in error.log
and see a lot of:
20010816T18:17:36: [notice] (elavil): bouncing a packet to
test_77 at elavil/Receiver from test_76 at elavil/Sender: Socket Error to
Client
20010816T18:17:36: [notice] (elavil): bouncing a packet to
test_77 at elavil/Receiver from test_76 at elavil/Sender: Socket Error to
Client
20010816T18:17:36: [notice] (elavil): bouncing a packet to
test_77 at elavil/Receiver from test_76 at elavil/Sender: Socket Error to
Client
20010816T18:17:36: [notice] (elavil): bouncing a packet to
test_77 at elavil/Receiver from test_76 at elavil/Sender: Socket Error to
Client
20010816T18:17:36: [notice] (elavil): bouncing a packet to
test_77 at elavil/Receiver from test_76 at elavil/Sender: Socket Error to
Client
...
This happens to several test_ users, not just test_76.
Ok.. so does this must mean the sockets are getting closed somewhere? I
checked both ends and none of my clients have reported a socket closed
condition. Must be something else. However, I do find a [very] few of
the following reported from msgloadsnd, which is catching bounce
messages:
[dpuryear at zoloft dpuryear]$ head s.test_76
msg: 7 997985807 901646 I think it is about time to say bye-bye to ICQ,
AIM, and MSN!
msg: Offline Storage
msg: Socket Error to Client
msg: Socket Error to Client
msg: 8 997985808 411677 I think it is about time to say bye-bye to ICQ,
AIM, and MSN!
msg: Offline Storage
msg: Socket Error to Client
msg: Socket Error to Client
msg: 9 997985808 921620 Are you serious? You want to do what with
Jabber?
msg: Offline Storage
I assume the "int int int string" messages are bounces? Regarding
"Offline Storage" I assume that just means jabberd is putting the
message into offline storage because it lost contact with the receiving
client, as I confirmed by looking at the test_77.xml file:
[root at elavil jabber-1.4.1]# ll spool/elavil/test_77.xml
-rw------- 1 jabber jabber 15110 Aug 16 13:19
spool/elavil/test_77.xml
(Odds are receivers and evens are senders, so test_76 sends to test_77.)
But notice that I got an Offline Storage message _before_ a Socket Error
to Client message. What happened there? And why isn't my receiving
client getting a socket closed condition? None of the clients ever
report that their socket has been closed.
Also, I don't always get warnings or notices from jabberd. For example,
examine the following:
20010816T18:24:06: [notice] (elavil): bouncing a packet to
test_391 at elavil/Receiver from test_390 at elavil/Sender: Socket Error to
Client
20010816T18:24:06: [notice] (elavil): bouncing a packet to
test_391 at elavil/Receiver from test_390 at elavil/Sender: Socket Error to
Client
But I have nothing in s.test_390, which contains stderr from my test_390
sending client:
[dpuryear at zoloft dpuryear]$ ll s.test_390
-rw-r--r-- 1 dpuryear dpuryear 0 Aug 16 13:16 s.test_390
So sometimes bounces get through, and sometimes not? Not sure what is
going on.
BTW, I ran jabberd on another machine and swapped the client machines
and get similar results.
Temas, thoughts? I archived all of the data from this test, including
the jabberd log files. Let me know if you want them.
Regards, Dustin
On 10 Aug 2001 11:56:55 -0500, Dustin Puryear wrote:
> FYI, I plan on building checks for bounced messages within a few days
> into the tools. Is this feature documented?
>
> On 09 Aug 2001 14:11:49 -0500, Thomas Muldowney wrote:
> > Yeah I think sheath is doing stuff with them. The reason I ask about bounces,
> > is because most of the queues that have set lenght or sizes will bounce back to
> > sender if they are overflowed (as far as I'm to understand). So although there
> > isn't guaranteed delivery, there is a response if it can't be. That's why I
> > say clients shouldn't need ACK's. Although IQ msgs are always discussed.
> >
> > --temas
> >
> > On Wed, Aug 08, 2001 at 02:51:34PM -0500, Dustin Puryear wrote:
> > > On 08 Aug 2001 12:16:08 -0500, Thomas Muldowney wrote:
> > > > Like I posted earlier I need more specifics to investigate this. Having clients
> > > > build in safeguards is ridiculous! This would be a server bug by far if no
> > >
> > > Well, I spoke with Colin (we are both with Vedalabs) about this and he
> > > mentioned that Jabber does not guarantee delivery. Is this true? If so
> > > then any application that requires all messages to be delivered will
> > > have to include some type of ACK feature.. I would think.
> > >
> > > > bounce is happening and it is actually getting lost somehow. It has high
> > > > priority in my books, but I need info, debug logs, incoming/outgoing XML for
> > > > both parties involved, and anything else that will help.
> > >
> > > I can do this. Note however that jabberd has to be under a high load for
> > > the problem to occur. That means you can expect _a lot_ of debug and log
> > > data. Perhaps you would like to run the test suite on a personal test
> > > box? It may be easier for you to see what data you need? Otherwise you
> > > can expect a 50MB email from me. :)
> > >
> > > Hmm, a while ago a guy from the OSDN asked for developer access to the
> > > testing tools. It would be nice if he could run it on his test boxes as
> > > well, but I'm not sure what happened to him.
> > >
> > > Regards, Dustin
> > >
> > >
> > >
> > > >
> > > > --temas
> > > >
> > > > On Thu, Aug 02, 2001 at 12:04:56PM -0500, Dustin Puryear wrote:
> > > > > On 29 Jul 2001 23:19:30 +0200, Gian Filippo Pinzari wrote:
> > > > > > > Notice that at >= 120 user pairs (240 connected users), which equates to
> > > > > > > 120 msg/sec in this test, my message loss rate varies from 3% to 13%.
> > > > > > > The average delivery also climbs to .14 seconds, but I don't consider
> > > > > > > that a problem. (However, the worst case delivery times are bad: > 6
> > > > > > > seconds for 150 and 160 user pairs.)
> > > > > > We've just developed a project (client+server) for one of the
> > > > > > biggest ISP in Italy. We found since the beginning that each
> > > > > > Jabber server was not able to handle more than a couple hundreds
> > > > > > client, so we implemented an architecture that load balances the
> > > > > > traffic among many concurrent servers running on the same or
> > > > > > different hosts. We also encountered message losses, but we didn't
> > > > > > care much :-). As Dustin, we thought this was due to client
> > > > > > problems.
> > > > >
> > > > > Well, now that I know someone else has had a similar experience I can
> > > > > only assume the problem is with jabberd. I had hoped that jabber.org
> > > > > would confirm this to either be a problem or something they had built
> > > > > into jabberd, but that information hasn't been very forthcoming.
> > > > >
> > > > > In the end I suppose developers using jabberd need to realize that it
> > > > > only works in low to medium load environments where message delivery is
> > > > > not a big issue. Unfortunately, I am not sure if the same is true of the
> > > > > jabber.com server since they didn't present any information on this
> > > > > topic either.
> > > > >
> > > > > Regards, Dustin
> > > > >
> > > > > --
> > > > > Dustin Puryear <dpuryear at usa.net>
> > > > > http://members.telocity.com/~dpuryear
> > > > > In the beginning the Universe was created.
> > > > > This has been widely regarded as a bad move. - Douglas Adams
> > > > >
> > > > > _______________________________________________
> > > > > jdev mailing list
> > > > > jdev at jabber.org
> > > > > http://mailman.jabber.org/listinfo/jdev
> > > --
> > > Dustin Puryear <dpuryear at usa.net>
> > > http://members.telocity.com/~dpuryear
> > > In the beginning the Universe was created.
> > > This has been widely regarded as a bad move. - Douglas Adams
> > >
> > > _______________________________________________
> > > jdev mailing list
> > > jdev at jabber.org
> > > http://mailman.jabber.org/listinfo/jdev
> --
> Dustin Puryear <dpuryear at usa.net>
> http://members.telocity.com/~dpuryear
> In the beginning the Universe was created.
> This has been widely regarded as a bad move. - Douglas Adams
>
> _______________________________________________
> jdev mailing list
> jdev at jabber.org
> http://mailman.jabber.org/listinfo/jdev
>
--
Dustin Puryear <dpuryear at usa.net>
http://members.telocity.com/~dpuryear
In the beginning the Universe was created.
This has been widely regarded as a bad move. - Douglas Adams
More information about the JDev
mailing list