Tag Archives: AMQP

[repost ]Website Monitoring with Distributed Messages/Tasks Processing (AMQP & RabbitMQ) on Django

original:http://www.slideshare.net/JimmyDeadcOde/website-monitoring-with-distributed-messagestasks-processing-amqp-rabbitmq-on-django

 

[repost]Product: Resque – GitHub’s Distrubuted Job Queue

original:

Product: Resque – GitHub’s Distrubuted Job Queue

Queuing work for processing in the background is a time tested scalability strategy. Queuing also happens to be one of those much needed tools where it easy enough to forge for your own that we see a lot of different versions made. Resque is GitHub’s take on a job queue and they’ve used it to process million and millions of jobs so far.

What is Resque?

Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

GitHub tried and considered many other systems: SQS, Starling, ActiveMessaging, BackgroundJob, DelayedJob, beanstalkd, AMQP,  and Kestrel, but found them all wanting in one way are another. The latency for SQS was too high. Others didn’t make full use of Ruby. Others still had a lot of overhead. Some didn’t have enough features. And still others weren’t reliable enough.

You may think GitHub is being too picky, but for GitHub background work is a core competency. Fifty percent of their work is background processing so they need a serious, robust, monitorable, performant queue system that meets their needs. These are exactly the circumstances when building your own makes perfect sense.

Resque is very full featured, especially for high volume sites, supporting: Persistence, See what’s pending, Modify pending jobs in-place, Tags, Priorities, Fast pushing and popping, See what workers are doing, See what workers have done, See failed jobs, Kill fat workers, Kill stale workers, Kill workers that are running too long, Keep Rails loaded / persistent workers, Distributed workers (run them on multiple machines), Workers can watch multiple (or all) tags, Don’t retry failed jobs, Don’t “release” failed jobs.

One innovation is to use Redis, a newgen key-value store instead of MySQL as the persistent store. A key element is the Sinatra-based based front-end for managing and monitoring the system. Often manageability is what separates simple libraries and servers from a complete, reliable system and Resque has taken that next step of professionalism.

Of course the downside of Resque is that it is Ruby centric and it’s yet another queue system, but it definitely looks to be worth a look if you are in the market for a solid production quality queue system.

Apache Qpid: Open Source AMQP Messaging

Enterprise Messaging systems let programs communicate by exchanging messages, much as people communicate by exchanging email. Unlike email, enterprise messaging systems provide guaranteed delivery, speed, security, and freedom from spam. Until recently, there was no open standard for Enterprise Messaging systems, so programmers either wrote their own, or used expensive proprietary systems.

AMQP Advanced Message Queuing Protocol is the first open standard for Enterprise Messaging. It is designed to support messaging for just about any distributed or business application. Routing can be configured flexibly, easily supporting common messaging paradigms like point-to-point, fanout, publish-subscribe, and request-response.

Apache Qpid implements the latest AMQP specification, providing transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support and a lot more. And Apache Qpid is extremely fast. Apache Qpid aims to be 100% AMQP Compliant.

AMQP Messaging Brokers

Qpid provides two AMQP messaging brokers:

  • Implemented in C++ – high performance, low latency, and RDMA support.
  • Implemented in Java – Fully JMS compliant, runs on any Java platform

Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP. See Download to see which messaging clients work with each broker.

AMQP Client APIs: C++, Java, JMS, Ruby, Python, and C#

Qpid provides AMQP Client APIs for the following languages:

  • C++
  • Java, fully conformant with JMS 1.1
  • C# .NET, 0-10 using WCF
  • Ruby
  • Python

Operating Systems and Platforms:

The Qpid C++ broker runs on the following operating systems:

  • Linux systems
  • Windows
  • Solaris (coming soon)

The Qpid Java broker runs on:

  • Any Java platform

Qpid clients can be run on the following operating systems and platforms:

  • Java:
    • any platform, production proven on Windows, Linux, Solaris
  • C++:
    • Linux
    • Windows
    • Solaris (coming soon)
  • C#
    • .NET

Getting Started

Getting Help

If you have a question about any aspect of Qpid or need help getting up and running please send an email to one of our mailing lists.

Getting Involved

We welcome contributions to Qpid. Mail us on one of our lists if you want to contribute to the project, have questions on using it or just want to get our thoughts on a topic…

Roadmap

For details on releases, a summary of what is in each release can be found here roadmap

Apache Qpid (Open Source AMQP Messaging)

Introduction

Enterprise Messaging systems let programs communicate by exchanging messages, much as people communicate by exchanging email. Unlike email, enterprise messaging systems provide guaranteed delivery, speed, security, and freedom from spam. Until recently, there was no open standard for Enterprise Messaging systems, so programmers either wrote their own, or used expensive proprietary systems.

AMQP Advanced Message Queuing Protocol is the first open standard for Enterprise Messaging. It is designed to support messaging for just about any distributed or business application. Routing can be configured flexibly, easily supporting common messaging paradigms like point-to-point, fanout, publish-subscribe, and request-response.

Apache Qpid implements the latest AMQP specification, providing transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support and a lot more. And Apache Qpid is extremely fast. Apache Qpid aims to be 100% AMQP Compliant.

AMQP Brokers

  • Java Implementation
  • C++ Implementation

AMQP Client APIs

  • Java (JMS 1.1 compliant)
  • C++ (Linux/Windows)
  • C# .NET
  • WCF Adapter (Windows Only)
  • Python
  • Ruby

Qpid Management

  • QMF Python API
  • QMF C++ API
  • Python tools
  • QMan (QMF<=>JMX)