What do message queuing services do

Design a scalable message queuing architecture


I recently started learning the nuances of scalable and enterprise computing architecture, and one of the key components is a messaging queue. To learn as much as possible from each programming paradigm, I am trying to implement my own version of a messaging queuing service.

So far, my initial design runs on a threaded socket listener, but to prevent the same message from being downloaded twice by two separate processing nodes, the message queue index register is locked when a read is initiated and unlocked after the register is freed updated. This eliminates the need for threading and means that there is an upper limit to the size of a scalable system based on the processing speed of the server on which the messaging queuing service is running.

The way around this is to run the message queuing service on multiple servers. However, this increases the likelihood that the same message will be downloaded twice. The only way to avoid such problems is to include an undo callback which (after the servers or even the threads on a single server have synchronized their information and detected such a reissue) instructs the Processing Node to stop executing its current job and poll the message queue again for the next message. However, there would be an upper limit at which most of the traffic sent would be syncs and revocation callbacks. This creates a bottleneck and slows down the processing of information so that a Many of the processing nodes would be performing zero operations and wasting time.

The final way to work around this problem is to have each Message Queue server (and each thread on each server) have a specific offset to determine where in the queue it is, especially when processing it must be done in a specific order.

So are there any message queuing architecture designs that could show me how existing enterprise message queuing services avoid these problems?






Reply:


In summary:

This is a serious problem. Don't reinvent the wheel.

There are many technologies that solve the message queuing layer. they include

  • ZeroMQ
  • RabbitMQ
  • Apache Kafka
  • Redis, with BLPOP or PUBSUB (I asked how that works).
  • Other AMQP implementations besides RabbitMQ

I think it is beyond my ability to discuss the disadvantages of each one, not least because I do not really claim the expertise to deal with them Cough good to do do not use rabbits to cough .

Even if you don't want to use any of these technologies, read their documentation.

In this way you get to know design patterns that are possible through a system. Reading the ZeroMQ documentation will teach you about many of the classic Message Queuing architectures that they are lovingly implemented. Even if you are not using ZeroMQ, you can use these patterns to evaluate other queuing technologies by asking if you can implement this pattern there.

Learn more about RabbitMQ / AMQP's Exchange Queue Model. Routing might be an option for you - this is supported by Redis PUBSUB, but I can't remember it being supported by ZeroMQ - and fanouts are used by my shop, even though they've been through a memcached poll (yuck!) Since have been poorly implemented for a while.

How do I choose one?

I work for a startup whose SLA is typical of a web app - some downtime is fine as long as we can quickly restore the service without data loss. We didn't have to think about scaling issues like we do with Twitter or Tumblr, so we didn't have to think about throughput volume either. However, when implementing an SLA similar to mine, the following considerations will come to mind:

  • The client libraries actually function ? Is it easy to keep a connection in them? (ZeroMQ, Redis: yes. RabbitMQ: no).
  • is it easy to monitor and manage from a server console? (Redis: yes, RabbitMQ: yes, ZeroMQ: not that I remember, but we haven't used it that long)
  • Do clients support internal queues so that there is little data loss in the event of brief outages? (ZeroMQ, Redis: yes. RabbitMQ: no.)

Of course, if you work for a high frequency trading shop, these are your less important concerns. You're more willing to invest development time in a client-side library to ultimately achieve higher throughput. But I'm writing this more to warn you that these technologies tend to be marketed for their performance rather than their out-of-the-box functionality. If you're a web startup, you care much more for the latter than the former, and accordingly something like Redis, optimized for ease of use when performing well than difficulty when performing well, is probably a better choice than RabbitMQ. (I really don't like RabbitMQ).




We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By continuing, you consent to our use of cookies and other tracking technologies and affirm you're at least 16 years old or have consent from a parent or guardian.

You can read details in our Cookie policy and Privacy policy.