[JDEV] asynchronous, efficient, in-band bytestreams / file transfer

[Darrell Berry]

hi

I understand from some googling that this is the kind of topic which may
rake over the ashes of ancient flamewars. 

I hope that's not the case: if there's a simple answer to the question,
so much the better!

---

I am developing a distributed system which involves the transfer of
*large* binary files. As, loosely, clients will wish to communicate
amongst themselves, under the supervision of a central authentication
and tasking server, I'm looking at various alternatives for MOM. The
issues are:

- clients need to be able to cope with asynchronous transfer -- some
will be offline at any given time.

- the protocol needs to work effectively through a wide range fo
firewalls and NAT environments. Pure p2p is ruled out by this (try
selling in firewall changes for pure p2p to a reluctant client IS
department!)

- the system will use a mix of point-to-point and pub/sub channels.

- the system must scale well. 

---

Jabber is nice. Very nice. And I can see that progress is being made on
issues related to inband data transfer (JEP-0047) (IBB) and the needs
for a firewall-friendly proxy (JEP-0003) (PASS). However both these
extensions seem to relate to synchronous transfer (i.e. both clients are
online and handshaking) -- although I'm not so sure that IBB won't queue
(can anyone confirm either way?)

Is there currently any way to use Jabber for to queue messages with a
significant payload size in a resource-efficient manner? As suggested
above, I want to avoid OBB to simplify the protocol, and to simplify
access through firewalls etc -- it is infeasible to have peers sharing
files directly over http or similar, and I would much prefer payload to
be included in messages than referred to by them!

---

If this seems too much like bending a perfectly good tool (Jabber) to a
task for which it is inappropriate, does anyone have any alternative
suggestions? I've been looking at xmlBlaster, which seems very flexible,
but lacks a persisitant backend store (hence queues live in memory --
not good when queued data may run into the range of Gb). Have also
looked at JMS, whcih seems capable of being bent into pretty much any
shape, but at the expense of much work. 

Jabber is well thought-out. But can it help with my needs? I'm asking
this on the DEV list solely becuase I need some input on your current
views on Jabber futures, but with a heavy implementation slant - i.e not
just good ideas, but ones that can actually be built!

thanks for any help

- darrell
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2544 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://www.jabber.org/jdev/attachments/20021109/259ec535/attachment-0002.bin>