Harness implemented an event-driven architecture using Redis Streams to enhance microservices communication, achieving faster delivery and scalability with minimal infrastructure changes, suitable for thousands of messages per minute.
The modern software industry is advancing rapidly. In a monolithic world, it was easy to put everything on the same machine, store the application state in a couple of databases (relational/non-relational), and scale servers horizontally. As the scale and the variety of use cases increased, the industry started adapting and introduced microservices - one service does a concentrated job and communicates just enough to other services to get the ball rolling.
How does state transfer happen with multiple services? One easy answer would be to make every service expose REST endpoints and make HTTP calls as and when required to the relevant services. This will definitely work well for some kinds of applications, but not all. If your application communicates synchronously (user-service as well as service-service communication), there is no need for additional complexity and you can just rely on HTTP calls. However, dealing with such a pattern can be challenging in the longer run for reasons like:
These are some of the issues which event-driven architecture fundamentally tries to address by making sure each service is doing what it is supposed to do and when it is supposed to do.
Event-driven architecture uses events to communicate among various independent microservices. It enables applications to act on these events as they occur.
There are various options available to implement event-driven architecture. Some of them are:
The most suitable one can be selected based on time to production, ease of adoption, message ordering, resilience, event replay, persistence, retries, at-least/at-most once delivery, cost etc of your use case.
We will discuss event-driven architecture at Harness using Redis Streams in this blog. Before diving into event-driven architecture, it is important to understand what Redis and Redis Streams are.
Redis is an open source in-memory data structure store, used as a database, cache, and message broker.
Important features of Redis:
Redis Streams is an append-only data structure that helps in various streaming use cases like real-time messaging (RTM), message broker, etc. Before diving into Redis Streams, let us first go through one of the other constructs which Redis provides for inter service communication: Redis Pub/Sub.
Redis Pub/Sub implements the Publish/Subscribe messaging paradigm. Let’s play around a bit with multiple terminal windows. Open four terminal tabs and enter redis-cli in all of them. We will PUBLISH to a channel from one window (Redis client) and SUBSCRIBE to that channel from the other three.
As you can see, the first client is publishing a message “hello world“ on channel1. The other three clients are endlessly subscribed to channel1 and receive the message sent via the publisher. A quick note here is that the subscribe command is a forever blocking operation and endlessly waits for a new message to appear on the channel.
Key things to note about Pub/Sub:
Redis Streams was introduced as a new data structure in Redis 5.0, which models an append-only log file like construct. Note the key difference between Redis Streams and Apache Kafka here is, Streams is merely an append-only list data structure in Redis with advanced operations, while on the other hand, Kafka is an entire platform of various components. Let us discuss the operations that we can perform on a Stream:
PUBLISH can be realized by XADD channel1 * message "hello world" and
SUBSCRIBE can be realized by XREAD BLOCK 5000 STREAMS channel1 $ and then keep passing the last fetched id in this loop.
Other commands like XPENDING, XCLAIM, XTRIM, etc. are discussed in the Redis Stream Intro page.
Key things to note about Streams:
We at Harness have recently moved to a microservices world for a coming release. Multiple services publish events, and the others selectively consume them by listening to a particular stream to drive end-user functionality.
I will give you a sneak peek into how easy it is to get it up and running with just a Redis instance and a few lines of code:
Here's a gist with the above code.
While implementing event-driven architecture at Harness using Redis Streams, these are some of the challenges that we faced:
We achieved an event-driven architecture by using Redis Streams and have deployed it to production for more than a few months now. Redis Streams is a more lightweight solution for implementing event-driven architecture, as compared to advanced solutions like Apache Kafka. You need to decide which would be the best implementation based on your use case and the features that you expect out of an event-driven architecture.
It seems you enjoy reading technical deep dives! How about reading our piece on Architecting Feature Flags for Performance at Scale, or Audit Trails 201: Technical Deep Dive?
Thanks for reading, and I hope you’re excited to use Redis Streams in your projects!
-Raj
Stop struggling with tools—master modern CI/CD and turn deployment headaches into smooth, automated workflows.