Event-Driven Architectures

In the real world, things happen, and we react to them, the same happens in any business, events are what keeps them running, be it a sale, a customer enquiry, a malicious attempt to breach your site or an error in one of your LOB applications. As in the real world, your business should react to these events, not all have the same priority or are important to the same audience in your company.

Albeit the last paragraph might sound like common sense for most people, a great majority of the architectures that businesses run on, don’t have the ability to share these events with all the parties that might have interest in them. Events are handled by the applications of a certain domain, for instance a sale is handled by the POS application, an invoice or a receipt is generated by that or another application of the same domain, but, if you need to inform your maintenance department that there is another product sold that they have to be aware of, it will occur on a batch process at night and only will be available tomorrow for them. If your marketing department wants to know which customers bought a certain product to create a new campaign, it will ask someone from the sales department that will ask someone from the IT department to run a query and get a report with that information. 

That is the beauty of event-driven architectures, you produce an event, and everyone who might have interest in it, will have the opportunity to react to it.

So, what is and Event-driven Architecture?

Event-driven architecture is design pattern where different, decoupled applications, asynchronously publish events to an event broker that can be subscribed and consumed. Once an event is sent, every application that has an access for that event type will be able to react to it. The application that sent the event does not need to be aware about which other applications will consume that event. The contrary is also true, a consumer does not need to know where an event originated.

How event driven architectures work?

In a simplistic way, there are event producers and event consumers, and there are two common ways of dealing with the events:

• Pub/Sub – The producers of events are publishers, and when they publish an event, it is made available to all the consumers that are enrolled as subscribers for that event type. If a new subscriber is enrolled, it will only have access to new events published after the enrolment.
• Event Streaming – All the events produced are stored in a log, ordered, and any consumer that has access to the stream can read all the events and is responsible for knowing which events already where consumed. In this way, a new consumer can replay past events created before it had access to the stream.

In both, the producer responsibility ends in the moment it delivers the event to the event broker. It does not know who will consume it, and its business logic can’t depend on the assumption that a certain consumer will be listening to consume it. This is a feature by design, even if one system is down, all the events addressed to it will be available when it comes online again, without a general stoppage of the entire solution.

When to use event-driven architecture?

There are some key scenarios where this kind of architecture is paramount:

• Microservices: Event-driven architectures are particularly well-suited for microservices-based systems because they enable different services to communicate and react to changes in other services asynchronously.
• Real-time analytics: With event-driven architectures, it is possible to process high-volume streams of data in near real-time, which is useful for applications such as real-time fraud detection, real-time customer behaviour tracking, and real-time inventory management.
• IoT: IoT systems generate vast amounts of sensor data which can be fed into an event-driven architecture for real-time processing and monitoring.
• Business Process Automation: Event-driven architectures can be used to automate business processes that span multiple systems, such as order processing, invoicing, and inventory management.
• Real-time monitoring: By using event-driven architecture, it's possible to monitor different parts of the systems and trigger alerts or other actions in real-time.
• Mobile app backends: An event-driven architecture can be useful when building mobile app backends because it enables different parts of the system to react to events in real-time, such as push notifications, location updates, or changes in user data.

In general, Event-driven architectures are a great fit when it comes to use cases that require near real-time processing, scalability, and flexibility. It can help different teams work in parallel and with decoupled services.

It doesn’t mean that it only makes sense on these scenarios, and even if you don’t have one of these scenarios at this moment, if your aim is growing your business, it will happen sooner than you think. And then, you’ll have a set of potential benefits that help you to respond more quickly to changing business conditions and to handle increased traffic and data volume as they scale. They can help to improve the overall performance of the system, to improve collaboration and to make the development process smoother.

If it’s fantastic, why aren’t every business using it?

Well, event driven architectures bring more overall system complexity and there are some challenges to overcome:

• Event schema management: With a distributed event-driven architecture, it can be challenging to maintain consistency and compatibility in the structure of the events that are being produced and consumed across different systems.
• Event versioning: When the producer of the event changes its schema, the consumers of the event needs to be updated accordingly. In a large-scale system this becomes very challenging.
• Testing: Because of the distributed nature of event-driven architectures, it can be difficult to test and validate the overall behaviour of the system, especially when dealing with real-time events.
• Security: As events flow through multiple systems, it can be difficult to ensure that they are properly secured and protected from unauthorized access or tampering.
• Variable Latency: The need to communicate across networks mixed with the time needed to actually process the events adds a variation on latency, which makes it a challenge to implement real-time applications.
• Idempotency and event ordering: In event-driven architecture, events can be delivered multiple times. Ensuring that the same event is not processed multiple times can be a difficult task.
• Debugging: When things go wrong in an event-driven architecture, it can be challenging to identify the root cause of the problem and fix it.

Although, these challenges may seem daunting, many modern tools, frameworks and libraries, can help to mitigate some of these issues and make event-driven architectures easier to implement and maintain.

Are there some tools to help me leverage it?

Yes, nowadays there are several tools that help you implement an Event-driven Architecture. This list is not extensive but has some of the more mainstream tools:

• Apache Kafka: An open-source event streaming platform that can handle large numbers of events and handle high throughput.
• Confluent: A popular platform for event streaming that is built on top of Apache Kafka. It includes additional features and capabilities such as a schema registry, REST proxy, and SQL query engine for Kafka.
• Azure Event Grid: A managed service provided by Microsoft Azure for event routing and processing.
• RabbitMQ: An open-source message broker that can be used to implement pub/sub or event streaming architectures.
• Apache Pulsar: A distributed pub-sub messaging system that supports multiple messaging models, including event streaming.
• Apache Storm: A distributed real-time computation system that can be used to process streams of data in real-time.
• AWS Kinesis: A managed service provided by Amazon Web Services for real-time processing of streaming data.
• Google Cloud Pub/Sub: A managed service provided by Google Cloud for real-time messaging and event streaming.
• Apache NiFi: An open-source tool for data integration and real-time data processing.
• Apache Flink: An open-source stream processing framework for distributed streaming data applications.
• Apache Samza: A stream processing framework that is designed to handle large numbers of events and handle high throughput.
• Solace PubSub+: A messaging middleware that enables the integration of different applications, services, and devices. The platform provides a wide range of features, including message queuing and publish-subscribe messaging.
• Red Panda: Real-time data streaming platform developed by a company called Red Panda. The platform enables developers to process, enrich, and analyse data streams in real-time. It is based on Apache Kafka and can be used to build complex event-driven applications, real-time analytics, and data pipelines.

It's important to mention that, depending on your use case, certain tools may be more suitable than others, and different tools can have different sets of features and capabilities.

Conclusion

In conclusion, event-driven architectures allow businesses to react to events in a flexible and scalable way, enabling them to respond to changing business needs and opportunities more quickly. By decoupling the producers and consumers of events, businesses can maintain greater agility and resilience, and avoid the risk of being slowed down by monolithic systems that are difficult to change. While implementing event-driven architectures can be challenging, the benefits of increased speed, flexibility, and scalability make it a valuable consideration for any business looking to grow and thrive in today's fast-paced, competitive environment.