[JDEV] File transfers
Max Metral
Max.Metral at PeoplepcHQ.com
Fri Jun 7 15:15:36 CDT 2002
Ok, this sounds more sane now, but I still want to make sure I'm
understanding you right. You say that:
"The bandwidth that goes through the Jabber server would have gone through
the pipe anyway."
In the case of using a direct client-to-client transfer (whatever protocol)
versus using a client-server-client transfer, the bandwidth flows and costs
are much different. Do you agree?
-----Original Message-----
From: Michael F Lin [mailto:MFLIN at us.ibm.com]
Sent: Friday, June 07, 2002 3:17 PM
To: jdev at jabber.org
Subject: RE: [JDEV] File transfers
The point of the model is that Jabber servers are deployed by ISPs for
their users, or corporations on their intranets. Again, just like sendmail.
The bandwidth that goes through the Jabber server would have gone through
the pipe anyway.
The null server idea is a fine idea. You are correct to say that "others"
have suggested it before. We brought it up about 6 months ago, to be
precise.
http://mailman.jabber.org/pipermail/jdev/2002-January/009653.html
http://mailman.jabber.org/pipermail/jdev/2002-January/009658.html
http://mailman.jabber.org/pipermail/jdev/2002-January/009963.html
(By the way, the idea is also copied from sendmail). Everything we are
suggesting here completely extends to null servers as well as regular
Jabber servers, and benefits them just as much. There are many other
powerful directions you can take the null server besides file transfer. It
is particularly nice for systemically defeating AOL's attempts to block us.
The trouble is that we need a solid, flexible, *open-source* native Windows
codebase (thanks, Tipic, but no thanks) for a Jabber server for this to
work. This is not something we are working on, for political reasons. It is
something that should be done eventually, and for this reason we are
keeping a close eye on Joe's Jabber.NET work. Unfortunately, Joe can't
build the Jabber server there, also for political reasons. But we can fill
in the infrastructure in the meantime.
-Mike
|---------+---------------------------->
| | Max Metral |
| | <Max.Metral at peopl|
| | epchq.com> |
| | Sent by: |
| | jdev-admin at jabber|
| | .org |
| | |
| | |
| | 06/07/2002 02:24 |
| | PM |
| | Please respond to|
| | jdev |
| | |
|---------+---------------------------->
>---------------------------------------------------------------------------
---------------------------------------------------|
|
|
| To: "'jdev at jabber.org'" <jdev at jabber.org>
|
| cc:
|
| Subject: RE: [JDEV] File transfers
|
|
|
|
|
>---------------------------------------------------------------------------
---------------------------------------------------|
It's not a size thing, it's an "existence" thing. In the case of
client-to-client, if I'm running a Jabber server at Exodus say, I pay for
pretty much 0 bandwidth for that transfer since that cost is paid by the
clients ISP's (which may or may not be me, most likely not). If the server
is involved, I pay for 2x the size of the file + whatever chunking or
retransmission overhead + temporary (how long) storage and resource load.
Now, *MAYBE* an answer could be that instead of HTTP as our "random"
protocol, we actually use a simple Jabber server on the client for
client-to-client, and that therefore if it happens to be hosted by the
"real" Jabber server, everything just works the same. Maybe this is what
others have suggested? But even then, I'd say we want a "parallel-band"
method rather than in-band, and I don't think it solves the H.323 and other
protocol problems...
-----Original Message-----
From: Michael F Lin [mailto:MFLIN at us.ibm.com]
Sent: Friday, June 07, 2002 1:46 PM
To: jdev at jabber.org
Subject: RE: [JDEV] File transfers
Max, I understand the argument about bandwidth costs if we were talking
about encoding all binary data in base64, increasing its size by 33%.
However, this does not need to be the case, as XIB and SOAP Attachments
(well, sorta) show. The model we envision for deployment of Jabber is just
like sendmail, with servers being deployed at the site level. This is the
model on which sendmail defeated propriety e-mail providers (online
services), and I propose it is the model upon which Jabber can defeat
proprietary IM providers. In this case there isn't much additional
bandwidth usage that you should have to pay for. Presumably, you pay for
your backbone connection, and the same amount of data is going to the
external Internet as would be the case in a P2P transfer to some external
Internet node. I'll grant there is some argument about increasing bandwidth
usage internally within your NOC (maybe over a culmulative 20 feet of
cable), but if you're not running at saturation this shouldn't cost you
anything, and if it does it's a one-time cost.
I am saying that file transfers and binary data of unextreme size or
bitrate belong in band. I don't have a concrete definition of "unextreme",
but I think we can come up with some examples:
Unextreme (in-band | parallel-band):
Office-type documents
Source files
PDFs
Pictures
Streaming audio (maybe)
Largish XML data islands which the server should not have to parse, load
into a DOM, etc.
Extreme for the forseeable future:
CD images
DivX
Extreme for the near future, but probably not for terribly long:
MP3s
Streaming video
The point is that it should not be necessary to jump through the absurd
hoops of negotiating firewalls with these stateful proxy servers and
talking and/or understanding a foreign protocol in order to transfer a
slightly largish chunk of data which does may or may not happen to be XML.
I am not saying that nothing should be done out of band. I am saying it is
feasible and very worthwhile to do certain things (which are more than what
we are doing now, and which will grow as time goes on) in-band.
I will agree that current Jabber technology is not suitable for this. We
would like to see a simple and ruthlessly efficient binary wire protocol
which uses XML exactly where XML is advantageous in JNG, and finally
acknowledges that the advantages of using XML as a wire protocol are not
the same as the advantages of using XML as a document format. Whether you
agree with all the large payload stuff or not, JNG needs a better wire
protocol, and I think everybody knows it. The point is that once you adopt
such a protocol (which allows the possibility of kernel-mode caching,
zero-copy streaming, page-aligned buffers, and other performance tricks),
not only do we get huge performance and scalability benefits, but
transporting large binary payloads becomes completely feasible - and it's
all almost as easy as what we're doing now. But more on that later.
-Mike
|---------+---------------------------->
| | Max Metral |
| | <Max.Metral at peopl|
| | epchq.com> |
| | Sent by: |
| | jdev-admin at jabber|
| | .org |
| | |
| | |
| | 06/07/2002 12:17 |
| | PM |
| | Please respond to|
| | jdev |
| | |
|---------+---------------------------->
>
---------------------------------------------------------------------------
---------------------------------------------------|
|
|
| To: "'jdev at jabber.org'" <jdev at jabber.org>
|
| cc:
|
| Subject: RE: [JDEV] File transfers
|
|
|
|
|
>
---------------------------------------------------------------------------
---------------------------------------------------|
Have you been looking at the fact that no service provider is going to be
able to afford a system that does in-band binary transfer exclusively? Are
you saying all file transfer and binary data (H.323) belong in band? Just
trying to make sure I understand your suggestion.
-----Original Message-----
From: Michael F Lin [mailto:MFLIN at us.ibm.com]
Sent: Friday, June 07, 2002 11:38 AM
To: jdev at jabber.org
Subject: Re: [JDEV] File transfers
I would also add that encoding concerns could be alleviated by using
something like Jer's XIB or SOAP attachments, which are extensions to XML
streams that allows inline binary payloads. There are a few things we would
do differently, but that's the general idea.
Again, the key advantage is that all packets are still JID-routed, so we
get through all firewalls and NATs without having to set up complicated
stateful proxy servers in addition to what we already have.
Really, the way to do this is to define a maximum _packet_ size, for all
Jabber packets. This makes everything manageable from the server side. If
you need to transfer a packet larger than that maximum size, either drop
out of band and deal with firewall problems yourself, or chunk the message.
Personally, I think the latter is preferable, because all these problems of
negotiating through firewalls and finding PASS servers just go away.
Unfortunately, it is not easy to define a maximum packet size because we
don't have length-prefixed framing, so you never know the length of a
packet until you've received the whole thing - by which time
you (almost) might as well send it along anyway! This is why we need
something like JEP-0017 or XATP.
Guys, frankly, we've been looking at this problem in this lab for the last
_year_ (some of you may remember me talking about it at JabberCon, over
most of the same objections). We've done the research and the prototypes
for high-performance Jabber-like messaging systems. I know the
disadvantages seem obvious and the advantages seem hard to accept - but we
are positive that high-performance, scalable in-band or parallel-band
transfer of large payloads is both practically advantageous and technically
feasible.
We are preparing a detailed proposal on the matter, which we hope will be a
major concrete step towards JNG. I hope we can soon put this silly argument
to rest for now.
-Mike
|---------+------------------------------>
| | "Michael Rothwell" |
| | <rothwell at holly-spr|
| | ings.nc.us> |
| | Sent by: |
| | jdev-admin at jabber.o|
| | rg |
| | |
| | |
| | 06/06/2002 04:23 PM|
| | Please respond to |
| | jdev |
| | |
|---------+------------------------------>
>
---------------------------------------------------------------------------
---------------------------------------------------|
|
|
| To: <jdev at jabber.org>
|
| cc:
|
| Subject: Re: [JDEV] File transfers
|
|
|
|
|
>
---------------------------------------------------------------------------
---------------------------------------------------|
The jabber server could simply have a config option for "max transfer
size," which ,when set to zero, disables all file transfers. Clients are
notified of the file-transfer capabilities of the server by the server. You
could even take it one step further and allow/disallow file-transfter use
per account ID and/or groups. Other config options which would be useful
are "cache time" and "max cache size" -- if cache time and/or size are
zero, then if the server cannot stream the data directly to the
recipient(s), it informs the client that the recipient isn't accepting
transfers. You could even limit the number of transfers/person/day, etc.
Leave it up to the clients to B64 encode/decode the transfer. The server
doesn't have to care, which will reduce processing load. The "max size"
parameters will be for the encoded versions, which is what server operators
care about, because it's what they pay for.
Server-supported transfers would be a nice config option, esp. for
behind-the-firewall and small-number-of-users servers.
----- Original Message -----
From: Gallo, Felix S.
To: 'jdev at jabber.org'
Sent: Thursday, June 06, 2002 3:54 PM
Subject: RE: [JDEV] File transfers
Mike writes:
> Firstly, there is no inherent problem with sending moderately
> large files through a software server. Sendmail does it all
> day, every day, on a massive scale, without relying on
> client-to-client connections.
However, most mail is accessed through POP, IMAP, or Exchange,
which are definitely client-to-client connections -- for the
simple reason that sendmail doesn't scale very well. For
every byte send to a sendmail server, two bytes traverse the
network. Most unfortunately, those two bytes always involve
just the one sendmail server and its attached network.
What positives do you get for having the sendmail server involved
in a large file transaction between two parties? You get a
guarantee of delivery and no need for continued storage on the
sending party's side (since the file 'moves' to the sendmail server).
If you're sending the file to multiple users at once, you get less
traffic on your local network (the server takes the load).
You also get an opportunity to mediate the file somehow (e.g.,
virus checking it as a service, converting it from aac into mp3,
storing it for later delivery to other users..)
What positives do you get for not having the sendmail server
involved? The network on the sendmail server sees 0X load
rather than 2X load; the latency is lower; the sendmail server
has no storage requirement; and you have arguably fewer points
of failure.
Pragmatically, taking the load off the server is more valuable
in the normal case than replicating HTTP/FTP/SMTP/FXP yet again.
The fact that 'SMTP does it' is not a great rationale for forcing
all the jabber servers to pay 2X bandwidth costs for file transfers
between their users.
F.
**********************************************************************
E-mail sent through the Internet is not secure. Western Asset therefore
recommends that you do not send any confidential or sensitive information
to
us via electronic mail, including social security numbers, account
numbers,
or personal identification numbers. Delivery, and or timely delivery of
Internet mail is not guaranteed. Western Asset therefore recommends that
you do not send time sensitive or action-oriented messages to us via
electronic mail.
**********************************************************************
_______________________________________________
jdev mailing list
jdev at jabber.org
http://mailman.jabber.org/listinfo/jdev
_______________________________________________
jdev mailing list
jdev at jabber.org
http://mailman.jabber.org/listinfo/jdev
_______________________________________________
jdev mailing list
jdev at jabber.org
http://mailman.jabber.org/listinfo/jdev
_______________________________________________
jdev mailing list
jdev at jabber.org
http://mailman.jabber.org/listinfo/jdev
_______________________________________________
jdev mailing list
jdev at jabber.org
http://mailman.jabber.org/listinfo/jdev
More information about the JDev
mailing list