Welcome to Distributed Event Streaming Platform Components. After watching this video, you will be able to: Describe what an Event is. List the common Event formats. Describe what an Event Stream Platform (ESP) is. List the main components of an ESP. And, list the popular ESPs. An event normally means something worth noticing is happening. In the context of event streaming, an event is a type of data which describes the entity’s observable state updates over time. For example, The GPS coordinates of a moving car. The temperature of a room. Blood pressure measurements of a patient, or a RAM usage of a running application. An Event, as a special type for data, has different formats. Let’s have a look at the three most common formats: It can be as a primitive type such as a plain text, number, or date, or an event may be in key-value format, and its value can be a primitive data type, or complex data type like list, tuple, JSON, XML, or even bytes. For example, the coordinates tuple of a car with car_id_1. Also, very often, an event can be associated with a timestamp to make it time-sensitive. Next, we will have a closer look at event streaming. Suppose we have one event source such as a group of sensors, a monitoring device, a database, or a running application. This event source may continuously generate a large event volume at a short time interval or nearly real-time. Those real-time events need to be properly transported to an event destination such as a file system, another external database, or an application. The continuous event transportation between an event source and an event destination is called event streaming. After all you have learned about ETL so far, you may think that to implement such an ETL process between one event source to one event destination should be straightforward. However, what if we have multiple different event sources and destinations? In a real-world scenario, event streaming can be really complicated with multiple distributed event sources and destinations, as data transfer pipelines may be based on different communication protocols such as: FTP: File Transfer Protocol HTTP: Hypertext Transfer Protocol JDBC: Java Database Connectivity SCP: Secure copy. And so on. An event destination can also be an event source simultaneously. For example, one application could receive an event stream and process it, then transport the processed results as an event stream to another destination. To overcome such a challenge of handling different event sources and destinations, we will need to employ the Event Stream Platform, or ESP. An ESP acts as a middle layer among various event sources and destinations and provides a unified interface for handling event-based ETL. As such, all event sources only need to send events to an ESP instead of sending them to the individual event destination. On the other side, event destinations only need to subscribe to an ESP, and just consume the events sent from the ESP instead of the individual event source. Different ESPs may have different architectures and components. Here we show you some common components included in most ESP systems. The first and foremost component is the Event broker, which is designed to receive and consume events. Since it is the core component of an ESP, we will explain it in more detail in the next slide. The second common component of an ESP is Event Storage, which is used for storing events being received from event sources. Accordingly, event destinations do not need to synchronize with event sources, and stored events can be retrieved at will. The third common component is the Analytic and Query Engine which is used for querying and analyzing the stored events. Let’s have a look at the Event broker, which is the core component of an ESP. It normally contains three sub-components: ingester, processor, and consumption. The ingester is designed to efficiently receive events from various event sources. The processor performs operations on data such as serializing and deserializing; compressing and decompressing; encryption and decryption; and so on. The consumption component retrieves events from event storage and efficiently distributes them to subscribed event destinations. There are many ESP solutions, including: Apache Kafka Amazon Kinesis Apache Flink Apache Spark Apache Storm and more such as Logstash in Elastic Stack, and so on. Each has its unique features and application scenarios. Among these ESPs, Apache Kafka is probably THE most popular one. In this video, you learned that: An event stream represents entities’ status updates over time. Common event formats include primitive data types, key-value, and key-value with a timestamp. An ESP is needed especially when there are multiple event sources and destinations. The main components of an ESP are Event broker, Event storage, Analytic and Query Engine. Apache Kafka is the most popular open source ESP. And finally, popular ESPs include Apache Kafka, Amazon Kinesis, Apache Flink, Apache Spark, Apache Storm, and more.
