[jdev] Jabber Certification Program

[Tijl Houtbeckers]

On Fri, 18 Jun 2004 13:33:20 -0400, Rachel Blackman  
<rcb at ceruleanstudios.com> wrote:

>>> The thing is, there are all these very cool Jabber featuresets out
>>> there, but lots of them are not necessarily supported.  Nor (other
>>> than peer pressure) is there much incentive for people to implement
>>> certain things.  I can look at Jabber and go 'wow, pubsub is a cool
>>> backend system, Stream Initiation will let me do a lot of really cool
>>> things down the line' and be excited, but your average IM user (for
>>> instance, my mother or father) will look at Jabber and go 'why can't
>>> I set a nice little picture like under MSN?  And why can't I use bold
>>> in my messages?' and so on.
>>
>> To me *that* sounds like you're saying: there are all kinds of cool
>> specifications out there, but noone is using them.
>
> I think I phrased this badly, so I'll try again in a moment to
> summarize my intent in this proposal.  However, I would submit that you
> are getting rather sidetracked by making this a debate over motives
> rather than benefits or drawbacks to this; regardless of /my/ intent in
> putting this proposal out there, surely you have your own opinions on
> what benefits or drawbacks this proposal has?  That was in large part
> what I was interested in hearing from others. ;)

Isn't the motive behind setting up a profiles specification or  
certification program important?

The worries I stated is that good ideas (certification and client  
profiles) can go bad if it drifts towards setting up a system that is  
simulair of entangled with the JEP process. Your text below confirms this  
in part.

> Okay.  Trying to summarize my intent again.
>
> Right now, there are lots of features out there.  To use XHTML as an
> example (or file transfer in the older days), because of the fact there
> is an attitude (rightly or wrongly) that implementation of experimental
> JEPs should be discouraged, sometimes we see custom extensions to
> clients as interim solutions.  (Witness various client-specific mutant
> HTML-marked-up message implementations.)

These (file transfer, XHTML) are in fact based on semi-standards from  
*before* the JEP process (simulair to iq:search, iq:agents, etc.) I don't  
think you should compare those with anything that's in or comes out of the  
current JEP process!

Avatars is a better example, this was actually one of the first JEPs. As  
far as I know implementations are compatible. They're just not implemented  
in every client that would be intrested in this because the way it's done  
is rejected by many client authors and the council.

So let's take this example and bring it to the certification proposal.  
What should we have done with avatars? Should it have been put in the  
certification when the spec was designed? Or when it was first implemented?
Should it ever have gone from "recommended" to "required" even though it  
never made DRAFT if many clients picked up on it (No, right?)? Now the JEP  
is RETRACTED, but there's still no alternative. Should you still recommend  
a RETRACTED JEP (from the JEP: " This JEP will not be considered further  
by the Jabber Software Foundation and should not be used as a basis for  
implementations.")??
If not, then wouldn't it have been strange to "recommend" a JEP for a  
feature for 3 years, and then have the whole feature itself removed as  
either recommendation or requirment? How will that help users?

The minute you start hand picking features while they're still in the  
middle of "emerging" through the JEP proces you're creating a parallel and  
incompatible structure.

> A certification program would:
>
> 1) Encourage standardization and interoperability, as anything
> 'certified' would be pretty much guaranteed to interoperate probably
> with the other certifications.

This point is absolutly valid. Though there still seems to be confusion on  
what excactly is proposed. Eg. certification on features vs. profiles  
(your reply to Matthew Millar seems to conflict with the earlier idea of  
profiles such as "minimal, intermidiate, extended). How deep would  
certification go, etc.

> 2) Indicate (through 'recommended' features in each year's
> certification definition) features which are not yet required for
> certification but which will likely be required in the following year.
> (Including experimental JEPs.)

However this point simply interferes with the JEP process, the "emerging"  
of features through the JEP process. Take the example of avatars.

> 3) 1 and 2 together; instead of spending effort making an 'interim
> solution' specific to your client while there is no clear
> non-experimental solution, all the client authors striving for
> certification and interested in a feature would thus have an incentive
> to concentrate their efforts on making a standard.  The JEP process is
> supposed to be driven not only by the Council but by the community, and
> this would drive community action.  Let's say someone wants a webcam
> solution in their client, but the only webcam JEP is Experimental and
> unfinished, not in a state where it can yet be implemented; if that
> webcam JEP is on the 'recommend' list for certification, then instead
> of going 'well, that JEP will not be Draft for a long time so I will
> just make my own client-specific solution for now,' the developer has
> an incentive to really push the JEP process forward, flesh out the JEP
> and so on.  Someone wanting certification is not required to implement
> the webcam JEP or participate in moving it forward, but might want to
> watch it if it is a potential for, say, the next year's 'Jabber Media
> Client' certification or whatever.

If I want to build a standerdized way of using webcams in Jabber, when you  
start the proces you know whatever you do it first, things *will* change,  
things *will* break, etc. You can also be pretty sure that if you put some  
time and energy in it in the end something will come out of the process  
that is useable.

Ofcourse inbetween specs and implementations will differ between different  
clients or factions. Don't try to fight this, it's part of the process of  
working to the final spec. What's the point of taking *one* of those specs  
and trying to recommend it to everyone? You *already* know people either  
don't agree on what to use or that the spec is still too broken to  
interoperate correctly.

> (This is not to say I think webcams will ever be a requirement on any
> certification, but I needed an example to pull from thin air.)
>
>> If we had this certification proces I don't think the number of
>> clients who would have implemented those obsolete ways would be that
>> much greater either. The very reason those ways are declared obsolete
>> is cause a large part of the developers do not agree with the methods
>> used. Nor would we have more clients who adopted the new ways (because
>> these are limited by others factors, the JEPs are still new, server
>> infrastructure is still missing (eg. IBB)), nor would it have sped up
>> the creation of new JEPs (having an installed base of the
>> old-now-obsolete experimental JEP that is also "certified" won't do
>> much good there!).
>
> To uses your example, let's say that the old avatar specification was
> 'certified' one year, and then it's found to be bad, and it's replaced.
>   Now you put the new avatar specification into the certification; those
> who want to keep their 'certified' status implement this.  Those who do
> not implement it do not get certification again.

See my questions on this example above :)

> I would say the problem is twofold.
>
> First, there is not a direct incentive to work on pushing JEPs forward
> -- implementing them, experimenting, finding their flaws and strengths
> and thus revising them so they mature -- and in fact (as has been seen
> in this thread), there are developers who actively /discourage/ people
> from implementing Experimental status JEPs, meaning they sit and
> languish.

I've heard *noone* says you shouldn't implement an experimental JEP.  
Justin K. didn't mean this either (as we speak there's a good chance he's  
doing *just that*, working on things like voice-chat, both JEPs and  
implementation!).

Let's get the JEP wording on here for those not familiar with it, the  
notorious EXPERIMENTAL JEP:

"Implementation of the protocol described herein is NOT RECOMMENDED except  
in an exploratory fashion (e.g., in a proof of concept). Production  
systems SHOULD NOT deploy implementations of this protocol until it  
advances to a status of Draft."

Does sticking certification labels on experimantal JEPs, placing those on  
websites for "Joe/Jane Average" users fall within the boundries of "proof  
of concept"? (could it be any more clearer??)

Like I said, are you really not distatsfied with the speed of the JEP  
process? (there have been discussion on this in the past). I don't think  
the solution is setting up a competing structure that is so intertwined  
with the process itself. I doubt it will be effective, I want to see how  
this would speed it up, but I don't see how.

> Secondly, there is no direct incentive to update to more current JEPs
> (save for interoperability) when ones go away.  With existing clients
> doing the old-style deferred avatar method, I will bet you that many
> will not consider it a priority to move to the newer avatar
> specification when it gets finished, which does hinder the adoption of
> newer protocol extensions.

And stamping "certified" on their old method for the past three years  
*will*?
Or do you think telling them for 3 years "this will be the way of the  
future!" and then when the next *experimental* way comes out telling them  
"no *this* will be it", only to be replaced with the next one etc. will  
help move the JEP process forward?

You only give an incentive to at one point implement a JEP that is not  
ready yet. Yes, ofcourse I do agree that certification of clients that use  
features for which the community has reached a concensus on what specs  
should be used is a Good Thing. But you only work against this by the  
recommendations you propose. Take the WebCam example.

I have a client, I want to build webcam stuff for it. So does someone  
else. So we start the proces of a JEP. Then others get the same idea. They  
help change our JEP or start on their own. To back our JEPs up we start  
making implementations. JEPs change, implementations change, bugs are  
found and fixed. etc. Eventually the council votes one of those specs to  
DRAFT. Up to then we all had our own implementations, some might work  
together, some might not. Then you hold up some pretty badges. Hey boys  
and girls, whoever implements the DRAFT spec will get one of these! That's  
good! Not only will we get a badge, all our different clients will work  
together, and you're there to make sure they do!

Now imagine, somewhere in the middle of that process, you come along with  
your badges. Ok, so maybe they're a little less shiny, but let's say 50%  
follows your "recommendation" and takes the current rev. implements it and  
puts it in their clients. Those other 25% might be standing by the side:  
"no we should fix the distribution so there's less network traffic.",  
"Look at the huuuuge security hole!", "It'll never work on OSX this way!".  
The other 50% says: "Yeah but it's only a recommendation, don't worry". So  
then 50% have a working implementation. It might be insecure, it might not  
work for everyone but it works for them. You're saying this is somehow an  
incentive to get the JEP moving till it works good and for everyone? Even  
if it means breaking compatibility for those 50% (and their users)? All  
cause you hold up a slightly shinier badge?

The JEP process, through the council structure, mailinglists etc. is our  
way in which we, as a community, try to come to a concencus on  
specifications for features. Yes it's slow, even frustrating and painfull  
at times, but the beauty of it is that the outcome is usually accepted by  
almost all involved, even if they are not convinced it's the best solution  
in their point of view. This is because all the energy and time put into  
this by all of us, that gives it a certain legitemacy..

What you propose is, at certain points in the middle of this process, let  
a person or worse, a "certification board" handpick a specification (from  
wich we don't know yet whether it is a sound spec or not, that's why we're  
still in the process, that's why we have the experimental implementations  
that break every rev. and we can use to look at things like performance  
and security, that's why we have the call for experience, etc) and  
"recommend" to as much people as possible to implement it for the end user!

I could be wrong here.. but then what *are* you recommending when you say  
"recommened"?

> In fact, this state of affairs encourages
> adoption of OLDER methods; if a new client author comes in, and wants
> to do avatars, and finds only one other client does the new pubsub
> method but five or six use the old method, I suspect the author in
> question is going to implement the old method as well.

I have nothing against certifying implementations of JEPs that have gone  
through the community process and come out. It's a fine idea, shown by the  
fact you're not the first to propose it or even do some work on it. To  
this you added the idea of making different profiles for clients. Some  
work on the basis of this has been done (such as Jabber basic client  
suite, by who other then PSA!). I've discussed it myself a few times  
privatly but I do think you're the first to bring it to the list (I could  
be ignorance on what was to follow but I my guess would be it's more what  
I lacked in that part, courage to face what you knew would be coming ;)

What I don't understand is how you think it would help features moving  
forward through the process by recommending or even certyfing them. I can  
see though, the temptation of doing this in order to get those features to  
your users, and have them somewhat work together with a few other clients.  
But is it really wise to do that?

Can you point me to a real world example of where they certify things that  
are still experimantal-community produced like this? Even DRAFT is far  
 from final in the jabber world..

The rest of the discussion is more about certification in general. We seem  
to agree a lot more there :)

>> Such a certification could have great benifits. Even if it won't speed
>> things up much, it would certainly enable the Jabber community to grow
>> as a whole (eg. look at the recent flare up in the PubSub discussion).
>> I would be carefull with what you call certification though, writing a
>> profile for clients and what they should implement is one thing,
>> checking wether a client entirely adheres to that specification *that*
>> could be called certification, but that is a difficult and costly
>> proces.
>
> Hence why you need the Certification Board to manage this, hence the
> original proposal.  And yes, it isn't that it could be "called"
> certification, that /is/ what was intended as certification. ;)

Just don't underestimate what real certification takes..
You'll never see me objecting to that though. :)

>> Just having the specifications for the different profiles (as agreed
>> by the major participants with an intrest of forming these specs) and
>> some organizing for interop. testing could still get you a lot of the
>> benefits! (As Stephen Pendleton suggested, real certification might
>> become an intresting commercial opportunity)
>
> Interoperability testing and standardized featuresets is a lot of the
> motive behind this, but my experience with other projects in the past
> is that something like a 'Certified' badge (and some kind of benefit to
> go with it) would drive the standardization and interoperability better
> than just saying 'hey, let's make it all work together.'  You catch
> more flies with honey than debate, or something like that, to mangle a
> metaphor.

I just wanted to point out that starting with organized interop. testing  
and client profiles would still get you a lot of benifts of certification,  
without going that far. (Hey, you have to *start* somewhere). And yes  
ofcourse you could have a badge or a logo for that. "my  
basic-IM-jabber-client survived the great 2005 interop. tests and all I  
got was this shiny badge" ;)

just my 0,01650438€..