Detect Absence of Events - WSO2 Siddhi Pattern

WSO2 Siddhi an opensource complex event processing engine which is used to power the WSO2 Analytics Server received a new feature from GSoC project: Non-Occurrence of Events for Siddhi Patterns. Until Siddhi 3.x, patterns can detect only the events that have arrived but building patterns based on events that have not arrived is an essential requirement and even there were questions on StackOverflow regarding this problem:
This article introduces the new pattern to detect absence of events along with its limitations and sample use cases. In order to get a clear idea, let's begin with a sample use case.
This feature is available from Siddhi v4.0.0-M50. Those who prefer stable version have to wait until Siddhi v4.0.0.
Suppose a grocery store wants to send flyers to the customers who have not visited their store for a week. To achieve this, a query using absent pattern can be defined as given below. For testing purposes, 1 sec interval is used instead of 7 days:
define stream CustomerStream (customerId string);

@info(name = 'query1')
from not CustomerStream for 1 sec
select *
insert into OutputStream;
After waiting for a second without sending any events to the CustomerStream, the following output will be emitted as the output.
Event{timestamp=1503330346312, data=[null], isExpired=false}
The output contains a single null attribute becuase the input stream has a single attribute: customerId and there is no event to get that id.

Ideally the Stream Processor should wait for infinite time to conclude that an event has not arrived. To avoid this problem, the waiting time is used so that if an event has not arrived within the waiting time, Stream Processor will assume that specific event has not arrived. There is a special case without the waiting time which will be discussed later in this article. The following table summarizes the possible basic absent event patterns along with the description.
not A for 1 sec Wait for 1 sec from the system startup; A should not arrive within that 1 sec waiting time
not A for 1 sec and B Wait for 1 sec from the system startup; A should not arrive within the waiting time and B should arrive at any time
not A for 1 sec and not B for 1 sec Wait for 1 sec from the system startup; both A and B should not arrive within the waiting time
not A for 1 sec or B Wait for 1 sec from the system startup; A should not arrive within the waiting time or B should arrive
not A for 1 sec or not B for 1 sec Wait for 1 sec from the system startup; either A or B should not arrive within the waiting time
A → not B for 1 sec Wait for 1 sec from the arrival of event A; B should not arrive within that 1 sec waiting time
P* → not A for 1 sec and B Wait for 1 sec from the arrival of event P; A should not arrive within the waiting time and B should arrive at any time
P* → not A for 1 sec and not B for 1 sec Wait for 1 sec from the arrival of event P; both A and B should not arrive within the waiting time
P* → not A for 1 sec or B Wait for 1 sec from the arrival of event P; A should not arrive within the waiting time or B should arrive at any time
P* → not A for 1 sec or not B for 1 sec Wait for 1 sec from the arrival of event P; either A or B should not arrive within the waiting time
not A for 1 sec → B Wait for 1 sec from the system startup; A should not arrive within that 1 sec waiting time and B sould arrive after 1 sec delay
not A for 1 sec and B → P* Wait for 1 sec from the system startup; A should not arrive within the waiting time and B should arrive at any time. After 1 sec delay and the arrival of B, P should arrive
not A for 1 sec and not B for 1 sec → P* Wait for 1 sec from the system startup; both A and B should not arrive within the waiting time. After 1 sec delay, P should arrive
not A for 1 sec or B → P* Wait for 1 sec from the system startup; A should not arrive within the waiting time or B should arrive. After either of them happened, P should arrive
not A for 1 sec or not B for 1 sec → P* Wait for 1 sec from the system startup; either A or B should not arrive within the waiting time. After 1 sec, either of them should not arrived and P should arrive
not A and B A should not arrive before B
A and not B B should not arrive before A
P* can be either a regular event pattern, an absent event pattern or a logical pattern

Restrictions & Recommendations

The absent event pattern has some restrictions at the query level due to its nature. The restrictions and recommended best practices are discussed in this section.
1: Absence Patterns cannot have reference id
Since there is no way to refer or select from an event that has not arrived, the NOT pattern cannot have the stream reference id (like e1=CustomerStream). Technically speaking, the following query is invalid and you cannot deploy it in Siddhi.
define stream CustomerStream (customerId string);

@info(name = 'query1')
from not e1=CustomerStream for 1 sec
select *
insert into OutputStream;

2: Absent pattern immediately following another absent pattern is not allowed
An absent pattern directly following another absent pattern is not allowed. Therefore, the following query is invalid.
define stream StockStream (symbol string, price float, volume int);

from not StockStream[price > 25.0f] for 1 sec -> not StockStream[price > 50.0f] for 1 sec
select *
insert into OutputStream;

3: Use constants to send meaningful output
Sending a constant instead of selecting everything can eliminate the null output problem. However, if an absent pattern is used with regular patterns, the attributes of the arrived events can be selected for the output stream:
define stream CustomerStream (customerId string);

@info(name = 'query1')
from not CustomerStream for 1 sec
select 'Customer has not arrived' as message
insert into OutputStream;

Sample Use Cases

Sample Use Case I
The hello world query provided in the begining of this article detects the customers who have not visited the grocery store for more than a week. Even though that query looks good, it is not useful when there are more than one customers who have not visited the grocery store for a week. To uniquely identify which customer has not visited the grocery store for a week, the query must be partitioned with the customerId as shown below. As you can see the query starts with the arrival of a customer and waits for a second and if the same customer has not arrived within a second, it will emit that customerId out.
define stream CustomerStream (customerId string);

partition with (customerId of CustomerStream)
begin
    from e1=CustomerStream -> not CustomerStream[customerId == e1.customerId] for 1 sec
    select e1.customerId
    insert into OutputStream;
end

Sample Use Case II
Sometimes, we need to find whether an event has arrived before the arrival of another event. For such purposes, a special type of absent pattern is defined using the AND operator. Assume we want to notify the customer to check Butter in his/her list when he/she purchases the Bread but not Butter. This can be done using a Siddhi query as given below which is using the absent pattern without a time frame. This query waits until a customer purchases Bread but if he/she has not already purchased Butter, it will generate an output.
define stream PurchaseStream (customerId string, item string);

partition with (customerId of PurchaseStream)
begin
    from e1=PurchaseStream[item == 'bread'] and not PurchaseStream[item == 'butter']
    select e1.customerId
    insert into OutputStream;
end
I personally find this as the most useful case because it does not depend on a waiting time. However the other cases listed in the table will be useful depending on the requirement. For more details about the absent pattern in Siddhi, feel free to comment below.
Previous
Next Post »

Contact Form

Name

Email *

Message *