[JDEV] japanese characters in yahoo transport

[Martin Shadbolt]

Hi All,

Background
=========
I am required to build a jabber client as a prototype to allow Japanese
chatting to any message system for an internal project in my company. I am
using java and swing as this supports Unicode and Japanese text entry can be
made. I am new to using Jabber.

 I can run my simple java client and connect to a jabber server (jabber.org)
as two different Users and communicate in Japanese ok.
>From the newsgroups I have read that the jabber server send XML packets
using UTF8 encoding so in my case the sequence is

Sending
---------
Enter Japanese  message text (Java Unicode) >>>> build XML message (Unicode)
>>>> send message (UTF8) >>> jabber server routes message

Receiving
-----------
Receive XML message (UTF8) >>>> decode XML message (Java Unicode) >>>>
display message ok


Problem
======
When I now try and speak to a Yahoo client or a Yahoo client try's to talk
to me the message in Japanese is corrupted.

 I have tried the following, which have failed

1. Taken the XML byte stream received from jabber server,
2. Convert from UTF8 (default encoding for jabber messages) ,
3. Display the Unicode message (incorrect display)

4. Taken the XML byte stream received from jabber server,
5. Convert from UTF8 (default encoding for jabber messages),
6. Taken Unicode chars in the message::body tag ,convert into "shift JIS
text
7. Display message (incorrect display)

My question is about the Yahoo transport. When it receives the native
message from Yahoo it builds the XML message. How does it then deal with the
bytes, which make up the message::body part?  Does it convert them into
Unicode? Just sent the chat message bytes, as is which implies the XML
message could have mixed encoding one for the xml file and one for the
message:: body tag?

If anybody has any suggestions on how I should encode/decode  the XML
message to send/receive messages to an external message system in Japanese
it would be a great help.

martin