[Social] Federating Social Networks (Supporting ServiceAdvertisements)

[Bob Wyman]

The results of last week's XMPP Summit are beginning to bleed out as Ralphm
blogs the first of a promised series of notes on the event.
See: http://ralphm.net/blog/2008/07/26/xmpp_summit_5
and http://ralphm.net/blog/2008/07/26/xmpp_social_networks_1

Not surprisingly, it seems that those at the Summit agreed that the most
sensible way to federate XMPP PubSub servers is to have various servers
subscribe to each other. Thus, if I was running a microblogging service that
provided open access to "public" posts on my service, I might set up a node
to which I published all such "public" posts. Other microblogging services,
search engines, etc. would then subscribe to that node and, by doing so,
could mix messages published to my service with those published to their own
service.

This approach of "Federation via Subscription" has some distinct advantages
over the alernative, "Federation via Publishing", particularly in that it
eases spam control and management of server resources. However, it has a
distinct disadvantage in that it makes it somewhat harder to form networks
of cooperating servers.

In a system which relies on Federation via Subscription, all servers that
receive messages must have knowledge of potential publishers prior to any
data flowing between them. Given two servers, A and B, no data will flow
from A to B unless B first becomes aware of A and subsequently subscribes to
at least one node on A. The interesting question becomes: "How does B become
aware of A?". Since no data can flow between the two servers until a
subscription is established, if there are no other mechanisms provided, one
must assume that B discovers A via "out-of-band" communications such as
email messages, phone calls, directory lookups, etc. These are, of course,
rather crude discovery methods and require manual configuration upon
discovery to establish federating subscriptions.

An alternative means for facilitating discovery would be to extend the
XEP-0060 PubSub specification to support a means for servers to publish
"Advertisements" which announce the availability of nodes for federation.
Advertisements would specify which nodes are available for federation and
what data will be published over those nodes. In order to reuse as much
existing framework as possible, Advertisements would be published just like
normal events, but they would be published to a "well known node" that is
commonly available on all services that support advertisements. This node
might be named: "http://jabber.org/protocol/pubsub#advertisements" and would
be like any other pubsub node in that it could be subscribed to, read, etc.
However, it would only support publishing <advertisement/>s not <event\>s.

The basic assumption behind federation is that two services will be
publishing data which is similar. For instance, that two micro-blogging
services will both be publishing micro-blogging entries that are formatted
as Atom entries. Agreement on the payload formats is essential to enable
federation. On the other hand, it is unreasonable to insist that all servers
use common node names. Thus, a mechanism is needed to provide a mapping from
some commonly agreed name for a stream of data and the node name that is
used on any particular server. This can be accomplished by having the
Advertisement provide a mapping from commonly understood logical node names
to local concrete names. Thus, those creating micro-blogging standards might
say that the logical node name for publishing public posts is:
http://example.com/PublicMicroBloggingPosts. Then, a server that published
public posts on a node named "987ye879799wwww00" would simply provide both
the local and logical name for the node in the advertisement.

Given this introduction, an advertisement might look like the following:
(but, use of an xdata form might be more appropriate and more flexible...)

<iq type='set'
from='new_service (AT) denmark (DOT) lit'
to='old_service.shakespeare.lit'
id='ad1'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>

<publish node='http://jabber.org/protocol/pubsub#advertisements'>
<advertisement xmlns='http://jabber.org/protocol/pubsub#advertisements'
id="*tag:new_service (AT) denmark (DOT) lit,2008-07-24:1234*">
<local node='987ye879799wwww00'
format='http://example.com/post_format'\>
<common node='http://example.com/PublicMicroBloggingPosts
<http://www.w3.org/2005/Atom>'\>
<description>All public posts on this server.</description>
</advertisement>
</publish>
</pubsub>
</iq>

If the Advertisement node is supported as a normal node, then it should be
possible for others to subscribe to the node and thus monitor advertisements
as they are published. Using filters, subscribers would either subscribe to
all advertisements published to the remote node or only to those
advertisements that are specific to that node. This permits advertisements
to flow to nodes not known to the advertiser as well as to permit servers to
ensure that they are rapidly made aware of changes to servers in which they
have an interest. Additional metadata such as keywords, etc. could be added
to make filtering easier and more effective.

Of course, "Advertisers" shouldn't expect that the mere act of advertising
will always result in a federating subscription. Server managers will still
often want to moderate the lists of nodes they subscribe to. Nonetheless,
the mechanism a foundation on which automatic subscription will sometimes
reasonably be built. For instance, I might wish to build a microblogging
aggregator that automatically subscribes to all remote services that claim
support for microblogging. Or, I might have a strong trust relationship with
some other service and decide that I would like to have my service subscribe
to anything advertised by the that service -- while manually reviewing
advertisements from other services... Many patterns are possible and
reasonable.

Those familar with blogging infrastructure will recognize a great deal of
similarity between the idea of Advertisements and that of "pinging." In
fact, within the blogging world, pinging is probably the most common and
useful means available to blog aggregators to discover new blogs. In fact,
it can be argued that the introduction of pinging and its use by blog
aggregators was probably one of the most essential steps in building the
blogging infrastructure as we know it today. Before pinging, the process of
discovering new blogs was horribly difficult, inaccurate and expensive for
service providers.

Comments? Does this sound reasonable?

bob wyman

[Ralph Meijer]

On Sat, Jul 26, 2008 at 03:46:51PM -0400, Bob Wyman wrote:
> The results of last week's XMPP Summit are beginning to bleed out as Ralphm
> blogs the first of a promised series of notes on the event.
> See: http://ralphm.net/blog/2008/07/26/xmpp_summit_5
> and http://ralphm.net/blog/2008/07/26/xmpp_social_networks_1
>
> Not surprisingly, it seems that those at the Summit agreed that the most
> sensible way to federate XMPP PubSub servers is to have various servers
> subscribe to each other. Thus, if I was running a microblogging service that
> provided open access to "public" posts on my service, I might set up a node to
> which I published all such "public" posts. Other microblogging services, search
> engines, etc. would then subscribe to that node and, by doing so, could mix
> messages published to my service with those published to their own service.
>
> This approach of "Federation via Subscription" has some distinct advantages
> over the alernative, "Federation via Publishing", particularly in that it eases
> spam control and management of server resources. However, it has a distinct
> disadvantage in that it makes it somewhat harder to form networks of
> cooperating servers.

You couldn't wait until the end of the series did you? ;-)

> In a system which relies on Federation via Subscription, all servers that
> receive messages must have knowledge of potential publishers prior to any data
> flowing between them. Given two servers, A and B, no data will flow from A to B
> unless B first becomes aware of A and subsequently subscribes to at least one
> node on A. The interesting question becomes: "How does B become aware of A?".
> Since no data can flow between the two servers until a subscription is
> established, if there are no other mechanisms provided, one must assume that B
> discovers A via "out-of-band" communications such as email messages, phone
> calls, directory lookups, etc. These are, of course, rather crude discovery
> methods and require manual configuration upon discovery to establish federating
> subscriptions.

The approach we wanted to take was to come up with the simplest possible
thing that could work. The federation via subscription model is actually
pretty much the as how we do federation in Jabber IM settings, and that
seems to work reasonably well, I would say. That said, we didn't stop
just were you seem to assume, although some have suggested it during the
summit. I won't outright dismiss what you suggest here, because I need
to take a better look at it, but we came up with a different way of
discovering pubsub nodes that I think is very natural to how the web
works.

The solution we came up with is using HTML and/or Atom link elements, on
pages/regular feeds that represent a person's updates, or his and and
his friends, etc. Pretty similar to how such pages currently point to
Atom/RSS feeds, we use a 'rel' of 'xmpp.feed' and the 'href' points to
the XMPP URI of the node, eg. xmpp:ralphm (AT) jaiku (DOT) com?;node=updates. This
provides for an easy way to have one user on service A to subscribe to
another user on service B: the first user is very likely to know about
the page of the second user at site B. He inputs that URL and the
service can do auto discovery and subscribe all behind the scenes. I'll
expand on it more on my blog, but that seems the gist of it.

I probably have more thoughts on this but need some sleep first. Cya!

--
Groetjes,

ralphm

[Lior Sion]

it sounds to me that using http to discover new updates will create heavy
stress on the server: tens of social networks each polling for new updates
all the time on all the combined users..

I might be missing something here, but if the publishing mechanism in
recursive (that is, I subscribe to twitter/user/ABC so I would get any new
thing related to ABC, their comments, tweets, presence, whatever) then what
will the new announcement mechanism add?

The difference between blog world pinging usages and social networking
federation is that in the blog world, there's a distinct one to many
relationship, or very close to it - many many small blogs updating a few
distinct aggregation services, while social federation is different, no?

Lior

On Sun, Jul 27, 2008 at 12:09 AM, Ralph Meijer <jabber.org (AT) ralphm (DOT) ik.nu>wrote:

> On Sat, Jul 26, 2008 at 03:46:51PM -0400, Bob Wyman wrote:
> > The results of last week's XMPP Summit are beginning to bleed out as
> Ralphm
> > blogs the first of a promised series of notes on the event.
> > See: http://ralphm.net/blog/2008/07/26/xmpp_summit_5
> > and http://ralphm.net/blog/2008/07/26/xmpp_social_networks_1
> >
> > Not surprisingly, it seems that those at the Summit agreed that the most
> > sensible way to federate XMPP PubSub servers is to have various servers
> > subscribe to each other. Thus, if I was running a microblogging service
> that
> > provided open access to "public" posts on my service, I might set up a
> node to
> > which I published all such "public" posts. Other microblogging services,
> search
> > engines, etc. would then subscribe to that node and, by doing so, could
> mix
> > messages published to my service with those published to their own
> service.
> >
> > This approach of "Federation via Subscription" has some distinct
> advantages
> > over the alernative, "Federation via Publishing", particularly in that it
> eases
> > spam control and management of server resources. However, it has a
> distinct
> > disadvantage in that it makes it somewhat harder to form networks of
> > cooperating servers.
>
> You couldn't wait until the end of the series did you? ;-)
>
> > In a system which relies on Federation via Subscription, all servers that
> > receive messages must have knowledge of potential publishers prior to any
> data
> > flowing between them. Given two servers, A and B, no data will flow from
> A to B
> > unless B first becomes aware of A and subsequently subscribes to at least
> one
> > node on A. The interesting question becomes: "How does B become aware of
> A?".
> > Since no data can flow between the two servers until a subscription is
> > established, if there are no other mechanisms provided, one must assume
> that B
> > discovers A via "out-of-band" communications such as email messages,
> phone
> > calls, directory lookups, etc. These are, of course, rather crude
> discovery
> > methods and require manual configuration upon discovery to establish
> federating
> > subscriptions.
>
> The approach we wanted to take was to come up with the simplest possible
> thing that could work. The federation via subscription model is actually
> pretty much the as how we do federation in Jabber IM settings, and that
> seems to work reasonably well, I would say. That said, we didn't stop
> just were you seem to assume, although some have suggested it during the
> summit. I won't outright dismiss what you suggest here, because I need
> to take a better look at it, but we came up with a different way of
> discovering pubsub nodes that I think is very natural to how the web
> works.
>
> The solution we came up with is using HTML and/or Atom link elements, on
> pages/regular feeds that represent a person's updates, or his and and
> his friends, etc. Pretty similar to how such pages currently point to
> Atom/RSS feeds, we use a 'rel' of 'xmpp.feed' and the 'href' points to
> the XMPP URI of the node, eg. xmpp:ralphm (AT) jaiku (DOT) com?;node=updates. This
> provides for an easy way to have one user on service A to subscribe to
> another user on service B: the first user is very likely to know about
> the page of the second user at site B. He inputs that URL and the
> service can do auto discovery and subscribe all behind the scenes. I'll
> expand on it more on my blog, but that seems the gist of it.
>
> I probably have more thoughts on this but need some sleep first. Cya!
>
> --
> Groetjes,
>
> ralphm
>



--
thanks,
Lior Sion

Skype: sionlior | GTalk: lior.sion

[Ayende Rahien]

I think that a more interesting question is what I am subscribing _to_.
Is this new service, or new users in known service?
That is, is this about getting information about new users in service I
trust, or about new services that pop up.

From the description, it sounds like it is something that is aimed at
discovering new services, not new items within existing service.

On Sun, Jul 27, 2008 at 12:17 AM, Lior Sion <lior.sion (AT) gmail (DOT) com> wrote:

> it sounds to me that using http to discover new updates will create heavy
> stress on the server: tens of social networks each polling for new updates
> all the time on all the combined users..
>
> I might be missing something here, but if the publishing mechanism in
> recursive (that is, I subscribe to twitter/user/ABC so I would get any new
> thing related to ABC, their comments, tweets, presence, whatever) then what
> will the new announcement mechanism add?
>
> The difference between blog world pinging usages and social networking
> federation is that in the blog world, there's a distinct one to many
> relationship, or very close to it - many many small blogs updating a few
> distinct aggregation services, while social federation is different, no?
>
> Lior
>
>
> On Sun, Jul 27, 2008 at 12:09 AM, Ralph Meijer <jabber.org (AT) ralphm (DOT) ik.nu>wrote:
>
>> On Sat, Jul 26, 2008 at 03:46:51PM -0400, Bob Wyman wrote:
>> > The results of last week's XMPP Summit are beginning to bleed out as
>> Ralphm
>> > blogs the first of a promised series of notes on the event.
>> > See: http://ralphm.net/blog/2008/07/26/xmpp_summit_5
>> > and http://ralphm.net/blog/2008/07/26/xmpp_social_networks_1
>> >
>> > Not surprisingly, it seems that those at the Summit agreed that the most
>> > sensible way to federate XMPP PubSub servers is to have various servers
>> > subscribe to each other. Thus, if I was running a microblogging service
>> that
>> > provided open access to "public" posts on my service, I might set up a
>> node to
>> > which I published all such "public" posts. Other microblogging services,
>> search
>> > engines, etc. would then subscribe to that node and, by doing so, could
>> mix
>> > messages published to my service with those published to their own
>> service.
>> >
>> > This approach of "Federation via Subscription" has some distinct
>> advantages
>> > over the alernative, "Federation via Publishing", particularly in that
>> it eases
>> > spam control and management of server resources. However, it has a
>> distinct
>> > disadvantage in that it makes it somewhat harder to form networks of
>> > cooperating servers.
>>
>> You couldn't wait until the end of the series did you? ;-)
>>
>> > In a system which relies on Federation via Subscription, all servers
>> that
>> > receive messages must have knowledge of potential publishers prior to
>> any data
>> > flowing between them. Given two servers, A and B, no data will flow from
>> A to B
>> > unless B first becomes aware of A and subsequently subscribes to at
>> least one
>> > node on A. The interesting question becomes: "How does B become aware of
>> A?".
>> > Since no data can flow between the two servers until a subscription is
>> > established, if there are no other mechanisms provided, one must assume
>> that B
>> > discovers A via "out-of-band" communications such as email messages,
>> phone
>> > calls, directory lookups, etc. These are, of course, rather crude
>> discovery
>> > methods and require manual configuration upon discovery to establish
>> federating
>> > subscriptions.
>>
>> The approach we wanted to take was to come up with the simplest possible
>> thing that could work. The federation via subscription model is actually
>> pretty much the as how we do federation in Jabber IM settings, and that
>> seems to work reasonably well, I would say. That said, we didn't stop
>> just were you seem to assume, although some have suggested it during the
>> summit. I won't outright dismiss what you suggest here, because I need
>> to take a better look at it, but we came up with a different way of
>> discovering pubsub nodes that I think is very natural to how the web
>> works.
>>
>> The solution we came up with is using HTML and/or Atom link elements, on
>> pages/regular feeds that represent a person's updates, or his and and
>> his friends, etc. Pretty similar to how such pages currently point to
>> Atom/RSS feeds, we use a 'rel' of 'xmpp.feed' and the 'href' points to
>> the XMPP URI of the node, eg. xmpp:ralphm (AT) jaiku (DOT) com?;node=updates. This
>> provides for an easy way to have one user on service A to subscribe to
>> another user on service B: the first user is very likely to know about
>> the page of the second user at site B. He inputs that URL and the
>> service can do auto discovery and subscribe all behind the scenes. I'll
>> expand on it more on my blog, but that seems the gist of it.
>>
>> I probably have more thoughts on this but need some sleep first. Cya!
>>
>> --
>> Groetjes,
>>
>> ralphm
>>
>
>
>
> --
> thanks,
> Lior Sion
>
> Skype: sionlior | GTalk: lior.sion
>

[Lachlan Hardy]

> it sounds to me that using http to discover new updates will create heavy
> stress on the server: tens of social networks each polling for new updates
> all the time on all the combined users..
>

I think he meant using HTTP to discover new sources to subscribe to and get
the updates from. Not using HTTP for the updates themselves.

Lachlan

[Blaine Cook]

I think the answer is simpler than discovery. Email, SMTP, is a favourite
example of mine. Discovery is limited to "who handles the mail for this
person, user (AT) example (DOT) com", and takes the form of a ubiquitous service (DNS)
at the lowest levels of the protocol, and thus only requires implementation
in core libraries.
If I want to share microblogging posts, photos, links, or any "social
object" in general, then I can do that. These services have been successful
in an asymmetric fashion. Although I'm interested in the BBC's photo stream,
it's unlikely they'll be as interested in my photo stream. In that sense,
the subscription model works well.

Moreover, It can work exactly like email, but in reverse --- I can tell my
grandmother that she can add me, "lattice (AT) flickr (DOT) com", to her photo site
(e.g., on facebook) as a friend/family member. In the background, the
process works exactly like email, though instead of sending an SMTP message,
the server sends a subscription request method. The interaction designers
take over, and give me a way to say "YES, I'd like my grandmother, "
rcook (AT) facebook (DOT) com" to see my photos." From then on, facebook.com will
receive a notification for "rcook (AT) facebook (DOT) com" any time I post a photo. The
only discovery required were two DNS lookups to find the XMPP servers.

That's it. That enables messaging from one site to another, which in turn
enables the rich interaction models that drive my interest in this
technology.

b.


On Sat, Jul 26, 2008 at 12:46 PM, Bob Wyman <bob (AT) wyman (DOT) us> wrote:

> The results of last week's XMPP Summit are beginning to bleed out as Ralphm
> blogs the first of a promised series of notes on the event.
> See: http://ralphm.net/blog/2008/07/26/xmpp_summit_5
> and http://ralphm.net/blog/2008/07/26/xmpp_social_networks_1
>
> Not surprisingly, it seems that those at the Summit agreed that the most
> sensible way to federate XMPP PubSub servers is to have various servers
> subscribe to each other. Thus, if I was running a microblogging service that
> provided open access to "public" posts on my service, I might set up a node
> to which I published all such "public" posts. Other microblogging services,
> search engines, etc. would then subscribe to that node and, by doing so,
> could mix messages published to my service with those published to their own
> service.
>
> This approach of "Federation via Subscription" has some distinct advantages
> over the alernative, "Federation via Publishing", particularly in that it
> eases spam control and management of server resources. However, it has a
> distinct disadvantage in that it makes it somewhat harder to form networks
> of cooperating servers.
>
> In a system which relies on Federation via Subscription, all servers that
> receive messages must have knowledge of potential publishers prior to any
> data flowing between them. Given two servers, A and B, no data will flow
> from A to B unless B first becomes aware of A and subsequently subscribes to
> at least one node on A. The interesting question becomes: "How does B become
> aware of A?". Since no data can flow between the two servers until a
> subscription is established, if there are no other mechanisms provided, one
> must assume that B discovers A via "out-of-band" communications such as
> email messages, phone calls, directory lookups, etc. These are, of course,
> rather crude discovery methods and require manual configuration upon
> discovery to establish federating subscriptions.
>
> An alternative means for facilitating discovery would be to extend the
> XEP-0060 PubSub specification to support a means for servers to publish
> "Advertisements" which announce the availability of nodes for federation.
> Advertisements would specify which nodes are available for federation and
> what data will be published over those nodes. In order to reuse as much
> existing framework as possible, Advertisements would be published just like
> normal events, but they would be published to a "well known node" that is
> commonly available on all services that support advertisements. This node
> might be named: "http://jabber.org/protocol/pubsub#advertisements" and
> would be like any other pubsub node in that it could be subscribed to, read,
> etc. However, it would only support publishing <advertisement/>s not
> <event\>s.
>
> The basic assumption behind federation is that two services will be
> publishing data which is similar. For instance, that two micro-blogging
> services will both be publishing micro-blogging entries that are formatted
> as Atom entries. Agreement on the payload formats is essential to enable
> federation. On the other hand, it is unreasonable to insist that all servers
> use common node names. Thus, a mechanism is needed to provide a mapping from
> some commonly agreed name for a stream of data and the node name that is
> used on any particular server. This can be accomplished by having the
> Advertisement provide a mapping from commonly understood logical node names
> to local concrete names. Thus, those creating micro-blogging standards might
> say that the logical node name for publishing public posts is:
> http://example.com/PublicMicroBloggingPosts. Then, a server that published
> public posts on a node named "987ye879799wwww00" would simply provide both
> the local and logical name for the node in the advertisement.
>
> Given this introduction, an advertisement might look like the following:
> (but, use of an xdata form might be more appropriate and more flexible...)
>
> <iq type='set'
>
> from='new_service (AT) denmark (DOT) lit'
> to='old_service.shakespeare.lit'
> id='ad1'>
> <pubsub xmlns='http://jabber.org/protocol/pubsub'>
>
> <publish node='http://jabber.org/protocol/pubsub#advertisements'>
> <advertisement xmlns='http://jabber.org/protocol/pubsub#advertisements'
>
> id="*tag:new_service (AT) denmark (DOT) lit,2008-07-24:1234*">
> <local node='987ye879799wwww00'
> format='http://example.com/post_format'\ <http://example.com/post_format'%5C>>
>
> <common node='http://example.com/PublicMicroBloggingPosts <http://www.w3.org/2005/Atom>'\>
> <description>All public posts on this server.</description>
>
> </advertisement>
> </publish>
> </pubsub>
> </iq>
>
> If the Advertisement node is supported as a normal node, then it should be
> possible for others to subscribe to the node and thus monitor advertisements
> as they are published. Using filters, subscribers would either subscribe to
> all advertisements published to the remote node or only to those
> advertisements that are specific to that node. This permits advertisements
> to flow to nodes not known to the advertiser as well as to permit servers to
> ensure that they are rapidly made aware of changes to servers in which they
> have an interest. Additional metadata such as keywords, etc. could be added
> to make filtering easier and more effective.
>
> Of course, "Advertisers" shouldn't expect that the mere act of advertising
> will always result in a federating subscription. Server managers will still
> often want to moderate the lists of nodes they subscribe to. Nonetheless,
> the mechanism a foundation on which automatic subscription will sometimes
> reasonably be built. For instance, I might wish to build a microblogging
> aggregator that automatically subscribes to all remote services that claim
> support for microblogging. Or, I might have a strong trust relationship with
> some other service and decide that I would like to have my service subscribe
> to anything advertised by the that service -- while manually reviewing
> advertisements from other services... Many patterns are possible and
> reasonable.
>
> Those familar with blogging infrastructure will recognize a great deal of
> similarity between the idea of Advertisements and that of "pinging." In
> fact, within the blogging world, pinging is probably the most common and
> useful means available to blog aggregators to discover new blogs. In fact,
> it can be argued that the introduction of pinging and its use by blog
> aggregators was probably one of the most essential steps in building the
> blogging infrastructure as we know it today. Before pinging, the process of
> discovering new blogs was horribly difficult, inaccurate and expensive for
> service providers.
>
> Comments? Does this sound reasonable?
>
> bob wyman
>
>

[Bob Wyman]

On Sat, Jul 26, 2008 at 5:09 PM, Ralph Meijer <jabber.org (AT) ralphm (DOT) ik.nu>wrote:
> You couldn't wait until the end of the series did you? ;-)
But, this subject is too much fun to wait for! Please do finish your series
quickly so that we can all get in the discussion!

> The solution we came up with is using HTML
> and/or Atom link elements, on pages/regular
> feeds that represent a person's updates, or his
> and and his friends, etc.
The problem you're solving here is different from and orthogonal to the one
I was discussing. Your solution is perfectly reasonable as a means to
discover an individual feed and would be particularly useful in finding
private feeds. However, it is not well suited to the needs of a service that
is trying to provide aggregation, search or discover services over public
feeds.

Today we have things like Summize and TweetBeep that provide search and
alerting services within the limited realm of public content posted to
Twitter. But these services rely on special relationships with Twitter.com.
That situation is very different from what we have in the more open world of
blogging where Google Blogsearch, Technorati, etc. are able to provide
services that span across all blogging sites due to the fact that blogs ping
and are thus easily discovered by the aggregators. As soon as a new blog
goes online, it pings one or more pinging services and immediately has its
public postings incorporated into the various services that do blog
aggregation, search etc.. This is, I think, a good thing. But, it would be
hard to do the same with a solution such as the one you propose (links in
HTML, etc.) if only because we can't be sure that all microbloggers would
actually have HTML or Atom feeds to carry those links and, even if they did,
it can take an awfully long time for an aggregator to find them.

The other problem I'm trying to address in my proposal is how one goes about
reducing the overhead of handling all the subscriptions that would exist in
a large social network where the number of subscribers to any particular
feed follows a power law based on the total number of potential subscribers.
There are issues of scale here... Imagine that I'm running an XMPP service
that has millions of users and you're running a small service that handles
several dozen users. Now, imagine that one of your users is very popular and
has 1 million of my users as subscribers. If only point-to-point
subscriptions are possible, then your service is going to have to provide
local storage for 1 million subscriptions and whenever your user publishes
data, your service is going to have to either send 1 million copies of the
message to my service or at least one copy with 1 million SHIM headers (see
XEP-0060). In either case, that's a lot of data! The alternative is for my
service to subscribe to your feed of all public data and then do the
filtering inside my own servers. (Yes, I should probably report back to you
on subscription counts and such... But, that's a different issue.) But, for
that to work, I need a way to discover not just a specific feed, but the
node which carries all public feeds on your server.

I'm not suggesting that a service would advertise something as granular as a
single user's feed or that it would advertise private feeds. Rather,
services should advertise things like "Feed of all public posts."

The advertisment mechanism is, in fact, one that is well understood in the
world of general publish/subscribe systems and would, of course, be used to
provide discovery for a variety of feeds including those that aren't
"social" per se. For instance, a service might advertise that it provides
"Weather Feed", "Stock Market Quotes", "Blog Posts", "Ski Resort
Conditions", "Movie Show Times", etc... that could be subscribed to,
filtered, etc.

bob wyman

[Bob Wyman]

On Sat, Jul 26, 2008 at 5:22 PM, Ayende Rahien <ayende (AT) ayende (DOT) com> wrote:
> From the description, it sounds like it is something
> that is aimed at discovering new services, not new
> items within existing service.
Yes. You are right. My primary intent in suggesting the Advertisement
mechanism is to allow the discovery of new services rather than individual
feeds within those services. And, as pointed out in another message, I'm
primarily concerned with the scale related issues that arise when trying to
provide cross-service aggregation or in serving large numbers of subscribers
that are spread between different services. That's a very different set of
issues than those that appear in discover of private, personal feeds.

bob wyman

[Ayende Rahien]

How many service providers do you expect to have?

On Sun, Jul 27, 2008 at 3:24 AM, Bob Wyman <bob (AT) wyman (DOT) us> wrote:

> On Sat, Jul 26, 2008 at 5:22 PM, Ayende Rahien <ayende (AT) ayende (DOT) com> wrote:
> > From the description, it sounds like it is something
> > that is aimed at discovering new services, not new
> > items within existing service.
> Yes. You are right. My primary intent in suggesting the Advertisement
> mechanism is to allow the discovery of new services rather than individual
> feeds within those services. And, as pointed out in another message, I'm
> primarily concerned with the scale related issues that arise when trying to
> provide cross-service aggregation or in serving large numbers of subscribers
> that are spread between different services. That's a very different set of
> issues than those that appear in discover of private, personal feeds.
>
> bob wyman
>
>

[Bob Wyman]

On Sat, Jul 26, 2008 at 9:13 PM, Ayende Rahien <ayende (AT) ayende (DOT) com> wrote:
> How many service providers do you expect to have?
It depends on the service. Aggregation is hard. Thus, I wouldn't expect
there to be a great many aggregators at any one time since aggregation is
hard -- especially if the aggregators do things like offer real-time
"tracking" services (prospective search). But, even if there aren't very
many, we've learned from blogging and the web that aggregators are very
important to the health of the overall system. Ideally, there would always
be more than one provider of any particular service to ensure competition
and thus ensure pressure to innovate by offering improved or new
capabilities. However, if we don't establish the protocols to enable
aggregation of social networking content, we're going to find that the only
service providers that have a chance to enter the market will be those that
have special relationships to existing major social networks -- for
instance, like the relationship between Summize and Twitter. The result, of
course, will be that it becomes impossible to compete with existing services
by innovating. We'll have a lock-in that will not serve the community's
interests.

It should be recognized that the "advertisement" mechanism has utility for
many kinds of publish/subscribe service -- whether or not they are
specifically related to social networking. Thus, the examples I gave in an
earlier note of "News Feeds", "Stock Market Quotes", "sports results,"
"weather reports", etc.. It is best, I think, if we try to have social
networking systems rely on technologies that are generally useful. We should
avoid unnecessary specialization so that the Social Networking space can
benefit from innovation and ideas that come from other realms.

bob wyman

[Ayende Rahien]

Aggregation is only useful if you have a lot of individuals participating.
If you don't, you can just form direct links to the publishers, without the
need for aggregation

On Sun, Jul 27, 2008 at 6:24 AM, Bob Wyman <bob (AT) wyman (DOT) us> wrote:

> On Sat, Jul 26, 2008 at 9:13 PM, Ayende Rahien <ayende (AT) ayende (DOT) com> wrote:
> > How many service providers do you expect to have?
> It depends on the service. Aggregation is hard. Thus, I wouldn't expect
> there to be a great many aggregators at any one time since aggregation is
> hard -- especially if the aggregators do things like offer real-time
> "tracking" services (prospective search). But, even if there aren't very
> many, we've learned from blogging and the web that aggregators are very
> important to the health of the overall system. Ideally, there would always
> be more than one provider of any particular service to ensure competition
> and thus ensure pressure to innovate by offering improved or new
> capabilities. However, if we don't establish the protocols to enable
> aggregation of social networking content, we're going to find that the only
> service providers that have a chance to enter the market will be those that
> have special relationships to existing major social networks -- for
> instance, like the relationship between Summize and Twitter. The result, of
> course, will be that it becomes impossible to compete with existing services
> by innovating. We'll have a lock-in that will not serve the community's
> interests.
>
> It should be recognized that the "advertisement" mechanism has utility for
> many kinds of publish/subscribe service -- whether or not they are
> specifically related to social networking. Thus, the examples I gave in an
> earlier note of "News Feeds", "Stock Market Quotes", "sports results,"
> "weather reports", etc.. It is best, I think, if we try to have social
> networking systems rely on technologies that are generally useful. We should
> avoid unnecessary specialization so that the Social Networking space can
> benefit from innovation and ideas that come from other realms.
>
> bob wyman
>
>

[Bob Wyman]

On Sat, Jul 26, 2008 at 11:33 PM, Ayende Rahien <ayende (AT) ayende (DOT) com> wrote:
> Aggregation is only useful if you have a lot of
> individuals participating. If you don't, you can
> just form direct links to the publishers, without
> the need for aggregation.

Of course. But, your comment makes me think I misunderstood your earlier
question. Let me clarify:

I think there will be a small number of major aggregation services relative
to the number of publishing services. The future should not be like today.
Today, we've only got a small number of services publishing data. But, in
the future, if we can get good protocols defined, we should have a great
number of them. The key reason we have only a small number of social network
services today is that we don't have support in the system for federating
data between services. The result is that the earliest services benefit from
"network effects" and lock out any new services. If we get the federation
protocols right, then it will be possible for dozens, if not hundreds, of
different services to be started and have a chance to prove themselves.

bob wyman

[Peter Saint-Andre]

Bob Wyman wrote:
> On Sat, Jul 26, 2008 at 5:09 PM, Ralph Meijer <jabber.org
> <http://jabber.org>@ralphm.ik.nu <http://ralphm.ik.nu>> wrote:
> > You couldn't wait until the end of the series did you? ;-)
> But, this subject is too much fun to wait for! Please do finish your
> series quickly so that we can all get in the discussion!
>
> > The solution we came up with is using HTML
> > and/or Atom link elements, on pages/regular
> > feeds that represent a person's updates, or his
> > and and his friends, etc.
> The problem you're solving here is different from and orthogonal to the
> one I was discussing. Your solution is perfectly reasonable as a means
> to discover an individual feed and would be particularly useful in
> finding private feeds. However, it is not well suited to the needs of a
> service that is trying to provide aggregation, search or discover
> services over public feeds.
>
> Today we have things like Summize and TweetBeep that provide search and
> alerting services within the limited realm of public content posted to
> Twitter. But these services rely on special relationships with
> Twitter.com. That situation is very different from what we have in the
> more open world of blogging where Google Blogsearch, Technorati, etc.
> are able to provide services that span across all blogging sites due to
> the fact that blogs ping and are thus easily discovered by the
> aggregators. As soon as a new blog goes online, it pings one or more
> pinging services and immediately has its public postings incorporated
> into the various services that do blog aggregation, search etc.. This
> is, I think, a good thing. But, it would be hard to do the same with a
> solution such as the one you propose (links in HTML, etc.) if only
> because we can't be sure that all microbloggers would actually have HTML
> or Atom feeds to carry those links and, even if they did, it can take an
> awfully long time for an aggregator to find them.
>
> The other problem I'm trying to address in my proposal is how one goes
> about reducing the overhead of handling all the subscriptions that would
> exist in a large social network where the number of subscribers to any
> particular feed follows a power law based on the total number of
> potential subscribers. There are issues of scale here... Imagine that
> I'm running an XMPP service that has millions of users and you're
> running a small service that handles several dozen users. Now, imagine
> that one of your users is very popular and has 1 million of my users as
> subscribers. If only point-to-point subscriptions are possible, then
> your service is going to have to provide local storage for 1 million
> subscriptions and whenever your user publishes data, your service is
> going to have to either send 1 million copies of the message to my
> service or at least one copy with 1 million SHIM headers (see XEP-0060).
> In either case, that's a lot of data! The alternative is for my service
> to subscribe to your feed of all public data and then do the filtering
> inside my own servers. (Yes, I should probably report back to you on
> subscription counts and such... But, that's a different issue.) But, for
> that to work, I need a way to discover not just a specific feed, but the
> node which carries all public feeds on your server.

As I understand it, what Ralph and Blaine and those guys came up with
while I was out of the room working on Jingle is that we could chain
pubsub nodes by having a "repeater node" at bigprovider.com subscribe to
that super-popular node at smallprovider.com. Clearly this works only
for open pubsub nodes (no fancy ACLs to be shared), but this approach
provides the simplest way to share information across services. We need
to define an ad-hoc command for subscribing mynode to yournode but that
should be pretty straightforward.

/psa