[JDEV] Some jabber architecture questions :-)

[Dennis Noordsij]

Hi,

I have to implement a new XDB system, ie get away from the xml-filesystem 
storage, and have already written several test apps connecting to jabber via 
tcp/ip but doing their own (p)threading (using libjabber and libxode still of 
course :). They work fine.

I have basically the framework for the different XDB components; I want some 
namespaces to go to a DB and some to the filesystem.

Basic idea is this: component receives and queues incoming xdb requests. 
Multiple threads process them (connection pooling, file caching, etc). 
Another thread writes xdb results back from an outqueue to jabber.

Question 1)
Which namespaces do I need to handle to completely remove the existing 
xdb_file.so and still maintain all functionality (aka "which namespaces go to 
xdb components").

So far I have:
	jabber:iq:auth
	jabber:iq:register
	jabber:iq:roster
	jabber:x:offline
	vcard-temp

Question 2)
Can the xdb replies be unserialized? Ie, half the requests go to a DB 
component, the other half to a(n enhanced) filesystem component. When jabber 
sends a request id 1, and then id 2, it might very well first receive the 
result for 2, and then the result for 1.

Because both components basically have a number of working threads a lookup 
for one request could take longer than another, but they happen in parallel 
and the first one to be finished is the first one to be returned to jabber, 
even if that was the last request received. (Basically to avoid ever have any 
part of the components block on something)

XDB results are therefore most likely completely out of order, although of 
course still within a reasonable time (hopefully few ms :-).

Code linked directly into jabber uses xdb_get and set, which block (please 
correct me if I am wrong). For performance reasons I don't want anything to 
block, that is why in an experiment program using the same framework when it 
had to do an xdb lookup it simply puts the related message (maybe the message 
that is supposed to be sent to offline storage) in a hashtable. Then a few ms 
later, after having handled maybe 10 other packets, an xdb result comes, and 
a different thread (any out of the pool) using the id the component fetches 
the message from the hash and sends the appropriate xdb type='set' xmlnode. 
This as opposed to having the thread wait for the reply (which also adds the 
overhead of making sure the right xdb result goes to the right thread that is 
waiting for it). 

Question 3) 
Not crucial, but more out of curiousity :-) How does jabber get affected when 
a module has to wait for an xdb result and it takes say 200ms to be fetched 
(think slow database or something). How much goes in parallel? (how many 
threads and how much serialisation?). How is progress on conversion to using 
pthreads (originally planned for release in late jan/early feb).



I have used this framework for other experiments, and the threaded packet 
processing works beautifully. I have all the necessary code to get the above 
to work, but stumble on the problems mentioned. I really hope to be able to 
get this going.


Hoping to receive some helpful pointers :-))
Kind regards,
Dennis