In Post 1, we reviewed Event-Driven Architecture (EDA) and message brokers. In Post 2, we explored four major messaging infrastructure in Azure: Azure Service Bus, Azure Event Hubs, Azure Event Grid, and Azure Queue Storage. In this post, we have analyzed 5 factors that can impact your decision making while choosing a cloud messaging broker.

Post 3: Decision Tree and Practical Examples

These factors are:

1. Event Volume

If you are implementing a solution with a low event volume you do not need an event streaming pipeline. You probably do not need topics and subscriptions either. A simple queue is probably sufficient.

If you are dealing with a high-volume scenario, depending on what type of messages or reliability requirements you have, there are different options. If your scenario involves a high volume of events such as telemetry or social media platforms, Event Hub is very suitable. Its ability to support HTTP and AMQP makes it a great candidate for many scenarios.

If you are dealing with rich high-volume messages such as payment requests, you need a service that offers high reliability (duplicate detection and dead-letter queue). Azure Service Bus is the best candidate for this situation.

If you plan to have a lot of messages in a single queue, the Queue Storage fits better because the size can grow beyond 80 GB.

2. Event Consumption Mode

All the services we reviewed support parallel consumers but in different ways.

Azure Service Bus supports competing consumers with a lock. Messages are deleted after receipt. Partitioning can be achieved with topics, subscriptions, and actions.

Azure Event Hubs is internally partitioned; it dictates to have parallel consuming applications under a consumer group. It is ideal to have one partition per receiver.

Azure Queue Storage service supports competing consumers with a lease. If not deleted, messages are restored in the queue.

Event Grid pushes messages to subscribers. Messages can be filters and sent to different subscribers by type, subject, or with other advanced filters.

3. Event Source

We can categorize events into two buckets: Azure resources events and application events. Resource events are generated when something happens to a resource. An event is produced when a file is uploaded to blob storage; an event is generated when a key expires in a Key Vault. If your events are generated in Azure, Event Grid is the clear candidate. If an Azure resource is not natively integrated with Event Grid yet, you will need to set up a custom topic or use another message broker.

Application events are produced by the application within Azure or outside of Azure. Employee creation within a web app is an example of an application event. IoT readings are also application events. Depending on the nature of the application events, Service Bus, Queue Storage, and Event Hubs can be used.

4. Message Size

The message size is not usually a big differentiator except in very specific scenarios. The message size for Queue Storage cannot be bigger than 64 KB. Azure Service Bus supports up to 256 KB with Standard tier and up to 1 MB for Premium tier.

Event Hubs and Event Grid support messages of size between 64 KB and 1 MB. Messages are billed in the increment of 64 KB.

If your messages contain large objects such as images, videos, and transactional data, you can use the claim-check pattern to protect the message bus and the client from being overwhelmed or slowed down. With the claim-check pattern, the message is broken into the claim and the payload. The claim is sent to a message broker service while the payload is sent to an external storage service such as a database or a blob storage resource.

5. Cost and Pricing Model: What do you pay for?

Cost is an important factor in deciding a cloud messaging broker. If you have enough budget, you will choose a solution that covers all your architecture requirements with a minimum effort. However, minimizing the cost is usually a requirement. Comparing the cost per event or message is not enough; understanding what you get charged for and for how much would help you choose the right message broker. For Queue Storage, you pay for storage, operations, and data transfer. Service Bus charges for operations brokered connections and an hourly fixed fee for standard and premium tiers. Event Grid charges operations and you also get 100,000 free events per month. Event Hubs charges for operations and throughput units (TU). You can also pay for event capture and extended retention.

Which factors are more important?

All the considerations we reviewed affect the final decision but not with the same weight. Volume, reliability, event source, and consumption mode drive the decision more than other considerations.

• Volume is the most significant consideration because it dictates the message broker architecture resiliency, throughput, scalability, and cost.
• The event source is very critical in choosing a message broker. If your events are generated by an Azure Resource; Event Grid is the clear candidate. In this scenario, other message brokers will be used as event handlers if needed. If your events are generated by the application, other criteria such as the volume, consumption, and reliability will drive the decision.
• Message reliability is important because some brokers such as queue storage do not handle errors very well natively. Azure Service Bus has more reliability features than others.
• The consumption model dictates how your application consumes events. The decision to pull instead of pushing, partitioning instead of competing influences the message broker decision.

A decision tree was developed using these four top considerations, but all other factors discussed in this section will influence your final implementation. In most cases, Event Grid cannot be used by itself; it is designed to seamlessly connect Azure resources. A bonus decision tree was developed to show how to integrate Event Grid with other message brokers and webhook-based resources such as Logic Apps and Functions.

Decision Tree

As shown on this decision tree, we have six different outcomes which are explained with examples in the next section.

Decision Tree Paths and Examples

Path (1) and Path (2) — Reactive Programming/Operations Automation:

• Scenario Setup

This path is applicable if your event source is an Azure resource that supports Event Grid natively (System Topic), you will implement Path(1). The list of system topics can be found here. This list is not exhaustive as Azure continues to integrate Event Grid into more resources. In this scenario, Event Grid shines because you only need to hook the event subscriptions. There is an increasing number of Event Grind Handlers. If the consumer is not a native event grid native handler, you can use a webhook to receive the message. Path (2) is similar to Path (1) except that the event source is not a native System topic, or it is an application outside of Azure. In this case, we need to use a custom topic that provides an endpoint where to send events. You also need to define the schema.

• Reference Architecture

Before we move to Path(3), we need to mention that Event Grid is not designed to retain messages like queue-based message brokers. Because of this limitation, Event Grid is usually used as a native connector between event producers and other message brokers such as Azure Service Bus, Event Hubs, and Queue Storage. We produced this bonus decision tree to illustrate how Event Grid connects with other message brokers for Path (1) and (2).

Path (3) — Workflow-based applications

• Scenario Setup

In this scenario, events are produced from an application or a system which is not an Azure Resource, there is a requirement to queue messages, and we are not looking for a full-blown sophisticated message broker with advanced features as we reviewed in Post 1. Another attraction for this path is the need to manipulate messages while they are still in the queue. If you are building a workflow-like application, this is a perfect cloud-native architecture. The azure team used this approach to build Durable Functions.

• Reference Architecture

Path (4) — High-Reliability Enterprise applications with a single consumer

• Scenario Setup

In this scenario, events are produced by applications; there is a need to queue messages until they are processed; high reliability is a requirement, and all messages are processed by one consumer or competing consumers without filtering.

• Reference Architecture

Path (5) — High-Reliability Enterprise applications with topics and subscriptions

• Scenario Setup

In this scenario, events are produced by applications; there is a need to queue messages until they are processed; high reliability is a requirement, and we need to route messages to different consumers based on some message characteristics. In this case, we need to use Service Bus with topics, subscriptions, rules, and actions if needed.

• Reference Architecture

Path (6) — Event Streaming

• Scenario Setup

In this scenario, we need to process millions of messages per second from different applications.

• Reference Architecture

Conclusion

You made it to the end of our Azure Messaging Solutions review. We reviewed Event-Driven Architecture and what it means for cloud distributed environments such as Azure. We evaluated four Azure services: Service Bus, Event Hubs, Event Grid, and Queue Storage. Using our technical understanding, we compared Azure message brokers using 5 considerations that drive the selection process. We used these factors to produce a decision-tree that can assist developers and solution architects in choosing a message solution in Azure.