[WS-XMPP] heavy load

[Roberto Ostinelli]

dear all,

i know this list has been unused for a while, and i sure do hope that all of
you guys are not too busy in using xmpp to implement m2m communications :) i
have an issue i'd like to discuss, if some kind soul around here will lend
an hear to me i'd be delighted.

we are using xmpp to send various data from xmpp clients to an unique jid.
this jid is a service, written in python, which needs quite some
computational power to respond. i need to ensure that this unique jid is
able to handle loads of traffic, and by loads, i mean *loads*.

i cannot use pubsub or chat rooms since we use <iq/> for request-response
mechanisms [pubsub and chat rooms only spreads out <message/> stanzas afaik,
unless i'm missing something here].

the option i am currently thinking is to use a custom server module [we use
ejabberd] that creates a route, and load balances incoming stanzas by
re-addresses them to different jids, each one of those being handled by a
different client [and hardware]. i.e., some kind of load balancer built in
ejabberd.

what do you think? how would you proceed? peter, is there something in the
protocol i'm missing out [definitely possible]?

i believe that taking care of havy traffic is in the very nature of m2m
communication, and this one shouldn't be too hard.

thank you in advance, xmpp-ers! :)

r.

[Greg Campbell]

On Fri, Oct 31, 2008 at 1:08 PM, Roberto Ostinelli <roberto (AT) openspime (DOT) com>wrote:

> dear all,
>
> i know this list has been unused for a while, and i sure do hope that all
> of
> you guys are not too busy in using xmpp to implement m2m communications :)
> i
> have an issue i'd like to discuss, if some kind soul around here will lend
> an hear to me i'd be delighted.
>
> we are using xmpp to send various data from xmpp clients to an unique jid.
> this jid is a service, written in python, which needs quite some
> computational power to respond. i need to ensure that this unique jid is
> able to handle loads of traffic, and by loads, i mean *loads*.
>
> i cannot use pubsub or chat rooms since we use <iq/> for request-response
> mechanisms [pubsub and chat rooms only spreads out <message/> stanzas
> afaik,
> unless i'm missing something here].
>
> the option i am currently thinking is to use a custom server module [we use
> ejabberd] that creates a route, and load balances incoming stanzas by
> re-addresses them to different jids, each one of those being handled by a
> different client [and hardware]. i.e., some kind of load balancer built in
> ejabberd.
>
> what do you think? how would you proceed? peter, is there something in the
> protocol i'm missing out [definitely possible]?
>
> i believe that taking care of havy traffic is in the very nature of m2m
> communication, and this one shouldn't be too hard.
>
> thank you in advance, xmpp-ers! :)
>
> r.
>
>
As long as you implement your service as a component (which should be
reasonably straightforward in most client libraries) rather than as a
standard client, ejabberd will automatically round-robin load-balance
stanzas among running instances of that component, which may suit your needs
out of the box. As with any component, you'll have to do your own
roster/presence management, of course.

-- Greg

[Roberto Ostinelli]

hi greg,

thank you for your fast response. i didn't know that ejabberd automatically
did load balance amongst components. this is great news. however,
unfortunately it won't be so immediate to implement it as service, since we
are using python libs that do not support components.


On Fri, Oct 31, 2008 at 9:39 PM, Greg Campbell <gtcampbell (AT) gmail (DOT) com> wrote:
>
>
> As long as you implement your service as a component (which should be
> reasonably straightforward in most client libraries) rather than as a
> standard client, ejabberd will automatically round-robin load-balance
> stanzas among running instances of that component, which may suit your needs
> out of the box. As with any component, you'll have to do your own
> roster/presence management, of course.
>
> -- Greg
>

[Egon Willighagen]

On Fri, Oct 31, 2008 at 9:08 PM, Roberto Ostinelli
<roberto (AT) openspime (DOT) com> wrote:
> i know this list has been unused for a while, and i sure do hope that all of
> you guys are not too busy in using xmpp to implement m2m communications :) i
> have an issue i'd like to discuss, if some kind soul around here will lend
> an hear to me i'd be delighted.

Johannes worked hard on a Java library that implements the IO-DATA
XEP, and I managed to get my own first service online using his
library just yesterday. So Roberto, excellent timing! :) See [0].

Are you using the IO-DATA XEP too?

Egon

0.http://chem-bla-ics.blogspot.com/2008/10/next-generation-asynchronous.html

--
----
http://chem-bla-ics.blogspot.com/

[Egon Willighagen]

Hi Greg,

On Fri, Oct 31, 2008 at 9:39 PM, Greg Campbell <gtcampbell (AT) gmail (DOT) com> wrote:
> On Fri, Oct 31, 2008 at 1:08 PM, Roberto Ostinelli <roberto (AT) openspime (DOT) com>
> As long as you implement your service as a component (which should be
> reasonably straightforward in most client libraries) rather than as a
> standard client, ejabberd will automatically round-robin load-balance
> stanzas among running instances of that component, which may suit your needs
> out of the box. As with any component, you'll have to do your own
> roster/presence management, of course.

I'm really new to the ejabberd daemon, but just created a 'component'
(see other reply in this thread).

So, apologies for asking such a newby question, but this sounds kind
of powerful... if the load on one machine (=component) becomes too
large, you just plugin in another one... I really like that idea. But
how would the ejabberd know which components are equivalent, and that
it can balance load over them?

Egon




--
----
http://chem-bla-ics.blogspot.com/

[Roberto Ostinelli]

hello egon!

On Sat, Nov 1, 2008 at 7:53 AM, Egon Willighagen <egon.willighagen (AT) gmail (DOT) com
> wrote:

> Johannes worked hard on a Java library that implements the IO-DATA
> XEP, and I managed to get my own first service online using his
> library just yesterday. So Roberto, excellent timing! :) See [0].
>
> Are you using the IO-DATA XEP too?
>

no, we are using openspime: www.openspime.org


On Sat, Nov 1, 2008 at 7:56 AM, Egon Willighagen <egon.willighagen (AT) gmail (DOT) com
> wrote:

> I'm really new to the ejabberd daemon, but just created a 'component'
> (see other reply in this thread).
>
> So, apologies for asking such a newby question, but this sounds kind
> of powerful... if the load on one machine (=component) becomes too
> large, you just plugin in another one... I really like that idea. But
> how would the ejabberd know which components are equivalent, and that
> it can balance load over them?
>
> Egon
>

hi egon,

it is indeed what greg suggested. load balancing is handled by the xmpp
server [and its clusters, i believe], calling the components that are
registered for a service. these are all 'trusted' components since they know
the same shared secret, thus are all registered for the same service.

cheers,

r.