[jdev] PubSub & News Feeds

[Kelly S]

Hiya.
 
I'm getting back into the jabber swing of things after a few years of being away and I'm digging into PubSub! 
 
I am planning to index a bunch of RSS/Atom etc via PubSub. I have read over some of the XEP and I am a bit confused about a few things so I would really appreciate some clarification :)
 
>From what I can read on PubSub its clear how users subscribe to feeds and receive events but how can I handle a user requesting a feed that is not indexed on the server yet? So that the server can now start indexing that feed and begin Publishing new entries it pulls (I will be writting the service which starts pulling feeds off the web).
 
Also a bunch of concerns pop into my head that I'm still unclear about when maintaining all this "feed" data. 1. Is there any way I can just publish the latest feeds I pulled down and pubsub discard any duplicates that may already exist? Or is the right way to handle this is to query every single entry individually to check if they exist before publishing them? 1.a If I have to query each individually. Would I make the entry id the url of the entry so that I can check for duplicates as I pull them down off the web and read the entries? (example 4 old entries, but 1 new entry since last pull). 2. Entry requesting. Is there any sort of querying we can use against them? If we are publishing tons of entries, someone may want to browse/read them but only request X amount at a time, or only newer than a certain date etc. I don't think pulling the entire entry history of a couple months is going to be too efficient. Some of my concerns for #2 is because we are going to mobilize this, data plans are expensive for mobile devices. In this country it can be $25/1.5MB/month. We want to try and make our mobile client query data efficiently but do these capabilities exist in Pub/Sub?
I'm sure I am just missing some concepts on how parts of PubSub work. 
Thanks so much for your time :)
_________________________________________________________________
Try Chicktionary, a game that tests how many words you can form from the letters given. Find this and more puzzles at Live Search Games!
http://g.msn.ca/ca55/207
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.jabber.org/jdev/attachments/20080524/ead9a092/attachment-0002.htm>