Events, which is just another word for messages, are a great way to communicate information between different systems. Apache Kafka is designed for handling these communications at a large scale. It has become an industry standard, in part because it provides an efficient way to get the data that we all have coming in to the where it can best be utilized.

In the same way that a language, like English, becomes valuable in large part because lots of people speak it, these messages are “understood” by most common systems and the ones that don’t can “be taught” without a ton of effort.

Just like this blog post is out there and we don’t have to even know about each other to share information, Kafka becomes your way of broadcasting events/messages to interested parties. And just like any spoken language, the more people that speak it, the more valuable it becomes. In a similar way, a shared event bus, like Kafka, that enables disparate systems to exchange events is even more powerful.

As an architect at Heroku helping our top tier customers, I have seen patterns that leave people and their companies trapped, and other patterns that make changes super easy. In this article I give a quick overview of five of the (many, many) patterns enabled by a standards-based event bus. While the patterns have different goals, down deep the implementation is basically the same, which keeps it easy, maintainable, and versatile.

This post is part of the series discussing event-driven architectures. Other posts in this series describe how to use Heroku compute to extend Flows, multi-org visibility for Work.com organizations, and cross-org data synchronization.

Heroku and Apache Kafka connectors are central to these patterns. If you are not familiar with either of them, you can check out the resources at the end of the post. (Also, Heroku Enterprise customers can request in-depth guidance on these topics from the Customer Solutions Architecture team.)

This post covers the following five patterns.

Pattern #1: Hey, did something change? (Change Data Capture)
Pattern #2: Better together! (Integration)
Pattern #3: We need to know now! (Real-time processing and insights)
Pattern #4: We might need that later. (Data archiving, batch processing)
Pattern #5: It is going to be HUGE! (Scalability)

Pattern #1: Hey, did something change? (Change Data Capture)

If your data never changes (which would be odd) you can skip this one, but you should probably go figure out why you can’t tell when it changes. In this pattern, the basic use case is that something changed in your Salesforce organization and lots of other services need to know about it. You don’t need to know what the services are or who owns them to enable the changes to propagate. (Of course, it’s important to keep security in mind.)

For example, let’s say you have an online store and when the inventory is updated in one of your regional Salesforce organizations you need to update the search for the site, the recommendations graph database, and the CDN. Lightning Web Components can also be updated directly from a shared Kafka event bus so that even apps running in your Salesforce organizations can be updated immediately after a change is made.

Timing is important to consider with this pattern as you don’t want to do something like recommending purchases for out-of-stock items. You will also want to make sure you have the ability to purge items too, also keeping in sync in the rollout. You have probably seen these types of issues when online shopping: An item is available but seems to disappear as you try to buy it. A more graceful way to handle it would be to mark the item as out of stock but offer a waitlist. It makes the customer feel like their time was not wasted and you don’t force them to go look elsewhere and lose the sale.

You could end up with an architecture like this:

Want to know more?

Heroku Connect
Using Apache Kafka on Heroku for syncing multiple Salesforce orgs (Webinar)
https://devcenter.heroku.com/articles/heroku-data-connectors
Elastic Search add-on
CDN add-on
Fastly + Kafka + Elasticsearch

Pattern #2: Better together (Integration)

Like peanut butter and jelly, coffee and donuts, or even copy and paste, many technologies go better together. The real strength of Apache Kafka is helping decouple systems while being able to share data between services or clouds. This is also another benefit of using a solution that is standards-based and open source. There are always new connectors and tools being built for Apache Kafka; you can use them as-is or use them to create your own. Ecosystems are stronger than products.

Sending events to the shared bus means that when you want to add a new service you can just connect it to the shared bus and things start working. In this diagram we connect to Kafka via PrivateLink, which enables it to be accessed securely like just another IP in the AWS VPC. We also have MuleSoft connected that then can be connected to many other services or APIs.

Want to know more?

Creating a PrivateLink for Apache Kafka on Heroku
Heroku Elements (Add-ons)
Mulesoft and Kafka connection example
Mulesoft connectors
Connecting Mulesoft with Heroku Data services

Pattern #3: We need to know right now! (Real-time processing and insights)

Some changes are time sensitive and you need to know about them as soon as possible. For example, consider a security breach, an item being shipped (or delivered), a service going down, or any other issue that needs action right now.

An architecture for analyzing real-time events and providing notifications or a dashboard might look like the diagram below. As the events come in on a topic, we process them and take action accordingly. We may be counting them and looking for an amount to trigger an action or simply showing those aggregates on a dashboard for monitoring.

Kafka Streams would be a reasonable way to implement this type of approach (see links below) and the architecture might look like this:

Want to know more?

Kafka Streams on Heroku
Kafka Streams architecture
Real-time uses in retail
Another real-time example
Managing event streams

Pattern #4: We might need that later. (Data archiving, batch processing)

So, you have all this data, now, where to store it?

Many companies have requirements to store data for a certain amount of time or store it offsite. Every company wants to know more about their customers and to gain insights from their data. This is a pattern for those folks (which is all of us, really)! Other than processing, another reason to store raw events somewhere is the ability to “replay” the events. You may need to replay them because of a bug or because you want to process the data in a new way. Whatever your use case, this is a pattern you will be glad to have implemented.

Want to know more?

Offsite database archiving
Tableau and Snowflake connection
Snowflake Kafka connector
Apache Kafka to Redshift Connector example app
Apache Kafka to S3, Google Cloud Storage, and more!
Apache Kafka, Redshift, Metabase example app
Replaying Kafka events example

Pattern #5: It is going to be HUGE (Scalability)

This thing is going to be huge!
When services are decoupled they become less fragile and more agile (more rhymes!). One thing that always slows down teams as the organization grows is “collaboration costs”, which is just a way of saying how much time you spend talking to other teams in order to get your team’s work done. So, by decoupling you can scale the application and the organization. Large monolithic apps can be slowly split into decoupled microservices that can be scaled independently. In the pattern shown here we are also scaling globally with Private Spaces in different geographic regions. We are taking the changes that come into Kafka and updating remote Redis caches in the other regions.

Want to know more?

Heroku Private Spaces
Heroku Redis
Event-Driven microservices with Apache Kafka
Kafka to Redis connector example

Conclusion

As you can see, these patterns are relevant for small companies connecting services as well as for large companies helping teams move faster (and not break things) as they scale up — or anywhere in-between. Once you have the pieces in place you will start seeing many other use cases. Kafka is a gateway to better architectures and after you get the basics wired up, the sky is the limit.

More great resources
Heroku Myths and Magic (free video course)
What is Apache Kafka?
Running Kafka Connectors on Heroku
Heroku Streaming Data Connectors launched!

[ If you are already a Heroku Enterprise customer, you can set up up a meeting here. Tell them Lyric sent ya! ]

About the Author:

Lyric Hartley is a Customer Solution Architect at Heroku. He spends his days helping Heroku’s biggest customers solve their hard problems and reach the scale they require. He loves that there are always new and difficult problems to solve. Before his current role he was on a development team at Salesforce.com. With an understanding from the inside of Salesforce, he helps many teams in Salesforce innovate faster with Heroku. He is passionate about the future of computing and enabling people to focus on what matters for their business instead of what is common to all businesses. He also loves music and beer!