Broker federation

Introduction

Imagine you are running a company in London. There's a lot of users and services there connected to OpenAMQ broker. Some time later you decide to expand and establish a branch in Paris.

While there are only few employees/services there, each can connect to the broker in London in a standard manner. However, as your Paris branch grows, the volume of data transferred between the offices grows. The message that's delivered to 100 clients in Paris actually passes the WAN 100 times.

Moreover, you have management and security issues. You have to take care of every computer in Paris and ensure it can create a connection to London-based broker.

At this point, what you want is to install a separate OpenAMQ broker in Paris and pass a subset of messages from London to Paris and vice versa. The users in Paris would then connect to the local broker via LAN facing little administration problems. Moreover, each message would be delivered from London once only and afterwards it'll be redistributed to Paris-based employees/services over the LAN.

This type of messaging architecture is called "broker federation". It is implemented in OpenAMQ using federations between exchanges (previously called 'message transfer agents' or MTAs). The picture below shows the problem (upper part of the picture) and how it can be solved (lower part):

mta1.png

Setup

OpenAMQ provides various ways to configure federations; automatically, at runtime, and manually in the configuration file. We'll use the manual configuration approach here.

To forward messages from London to Paris add the following section to your amq_server.cfg configuration file on the Paris-based box :

<federate
    exchange = "market-data-lse"
    attach = "openamq-broker-london-001"
    type = "publisher"
    login = "peering"
    vhost = "/"
/>

See the OpenAMQ documentation on federation for details on the <federate> configuration values.

This setup is sufficient to pass messages from London to Paris. Note that no specific setup has to be made in London. Actually, London broker doesn't even have to be restarted. Also note that:

  • Each message is passed over the WAN once only, even if there are 100 subscribers for it in Paris.
  • If there are no subscribers for a message on the remote location (Paris) the message won't be passed over the WAN at all.

So, when there are no users connected to the Paris broker (say at night) the link between London and Paris is completely idle (with the insignificant exception of heartbeats) irrespectively of what amount of messages is being passed in London.

At the moment, the messages are passed in single direction only - from London to Paris. Now, how can we make the message sharing work in both ways?

The obvious solution is to edit the configuration file in London the same way as we did in Paris:

<federate
    exchange = "market-data-euronext-paris"
    attach = "openamq-broker-paris-001"
    type = "publisher"
    login = "peering"
    vhost = "/"
/>

This is the preferred way to interconnect two brokers. However, there may be several reasons that can make editing London configuration file impossible:

  • The London broker may be run 24/7 with no way to restart it for the new configuration file to take effect.
  • The London broker may belong to a different company. In that case you have no privileges to configure their broker.
  • The broker in London may be AMQP compliant, but not OpenAMQ itself. In that case there is no federation mechanism available there.

In one of the above is true, you can still pass data from Paris to London, although in a less efficient way. You can add a new federation to the Paris broker, specifying the "publisher" type. The configuration file in Paris would thus contain two federation definitions:

<federate
    exchange = "market-data-lse"
    attach = "openamq-broker-london-001"
    type = "publisher"
    login = "peering"
    vhost = "/"
/>

<federate
    exchange = "market-data-euronext-paris"
    attach = "openamq-broker-london-001"
    type = "subscriber"
    login = "peering"
    vhost = "/"
/>

The drawback of the publisher federation is that it passes all the messages to the remote broker. So, even if there is no subscriber for a message in London, the message will be still passed from Paris to London and then dropped by the London broker. Still, if there are multiple consumers for the message in London, it will be passed once only over the WAN.

There are other federation types, please refer to the OpenAMQ documentation. For example:

  • The "locator" federation passes only those messages that have no consumers on the local broker to the remote one. The idea is that on SOA-based system (have a look at a tutorial here) you would like to request a work to be done by a local service, however, if the service is not available locally, the request will be forwarded to the remote broker. So, for example, if order management system in Paris fails, the traders in Paris would failover to the order management service running in London.
  • The "fanout" federation passes messages on particular exchange from London to Paris and vice versa. You cannot achieve this behaviour by combining publisher and subscriber federations as every message would be circulated endlessly between Paris and London. However, note that this federation is not particularly efficient. For best performance create two exchanges (market-data-lse and market-data-euronext-paris) instead of single one (market-data) and pass messages only in appropriate directions.

Extending the federation

As your company grows you establish branches in Berlin and Hong-Kong and connect them to the London in the same way as Paris broker is. At that moment you decide to expand to mainland China and establish branch offices in Beijing and Shanghai. It happens that the connection from London is slow, low-bandwidth or too expensive. Therefore you would like brokers in your mainland offices to speak to the broker on your Hong-Kong site rather than directly to the broker in London. Have a look at the diagram bellow. Note that messages are delivered in a slower manner - there are 2 network hops involved to pass a message from London to Shanghai - however, the lower intercontinental bandwidth usage etc. may still pay off.

mta2.png

You can define various advanced setups for your company. Recall that federation is done at the exchange level. Therefore, if your traffic is reasonably split across several exchanges, you can define message distribution network for each exchange separately.

Say, there is a trading application that need lowest latency possible: You should establish direct links between individual locations to eliminate excessive network hops.

However, there is also a post-trade reporting service that doesn't need to be that quick: You can pass the messages via a hierarchical tree-like broker federation.

Both distribution schemes can be used at the same time using same broker instances provided that message traffic is clearly separated into "trading" and "reporting" exchanges. This kind of setup can be seen on the diagram bellow. "Trading" federations are represented by full lines, "reporting" federations are represented by dashed lines (Hong-Kong and Paris report directly to London, Berlin reports indirectly via Paris):

mta3.png

Comments

Add a New Comment

Edit | Files | Tags | Source | Print

rating: +4+x

Author

Martin Sustrik <moc.xitami|kirtsus#moc.xitami|kirtsus>
Pieter Hintjens <moc.xitami|hp#moc.xitami|hp>

All tutorials

Performance Tests: This tutorial explains how to do OpenAMQ performance tests using 0MQ performance testing framework.

Broker federation: How to setup a geographically distributed federation of OpenAMQ brokers and how to define dataflows between individual geographical locations.

The ESB cookbook: How to implement your own Enterprise Service Bus, step by step, using OpenAMQ.

Publishing contents: How to publish contents at high speed without bizarre memory issues.

Tuning OpenAMQ for low latency: How to tune your operating system and hardware to get the lowest latency, i.e. the best response times, from your OpenAMQ broker.

The mandatory and immediate flags: How to use the 'mandatory' and 'immediate' flags on the Basic.Publish method.

Load balancing: How to use OpenAMQ to distribute work between multiple applications, a technique called "load balancing".

Content based routing: How to route messages based on their header properties. It is a fast and simple way to do 'content based routing' without needing to inspect entire messages. We explain the principles and provide worked examples.

Transient or durable subscriptions: How to make subscriptions that are transient (come and go with their consuming applications) or durable (stay around).

The AMQ model: How the AMQ model works: this is a basic backgrounder for anyone starting to use OpenAMQ in their applications.

Developing on Windows: How to build OpenAMQ client applications using MSVC on Windows

Handling Ctrl-C in applications: How to properly detect an interrupt (Ctrl-C) and shut-down gracefully in C/C++ WireAPI applications.