GitHub Pull Request - A Complete Guide For Beginners

The power of open source applications highly depends on the contributions from the community. As a maintainer of two small open source applications, I have seen the power of such contributions. However, there are people who like to contribute but fail because of the technical barriers. This tutorial shows you a top to bottom step by step guidance on how to add a new feature to an open source application that is available on GitHub.

Safe Eyes is one of my open source application available for Linux users to prevent asthenopia (eye strain) due to RSI. This tutorial shows you how to contribute in updating the French translation of this application by sending a pull request.

Prerequisite:

  • GitHub account
  • Git (Install Git)
  • Atom text editor (You can use any text editors but I prefer Atom due to its support for GitHub)
Step 1: Login to your GitHub account and then visit the GitHub repository of your interested open source application. In this tutorial, I use https://github.com/slgobinath/SafeEyes.


Read More

WSO2 DAS - Hello World!

WSO2 Data Analytics Server is a smart analytics platform for both real-time and batch analytics. The real-time analytics is provided through their powerful open source Complex Event Processing Engine engine Siddhi. This article focuses on the complex event processing capability of the DAS server and provides a quick start guide on how to setup an event stream and process events generated by an HTTP source.

WSO2 DAS - Hello World!

Prerequisite:
Oracle JDK 1.7 or latest
Memory – Minimum 2GB
Disk – Minimum 1 GB
For more details: Installation Prerequisites

Read More

Is WSO2 CEP Dead? No! Here’s Why…

During the WSO2 Con US 2017, a major business decision is announced. Due to some business decisions, WSO2 promotes the Data Analytic Server (DAS) (They may change this name very soon) over the complex event processor. For those who haven’t heard about DAS even though it has been there for a long period, it is another product of WSO2 which contains the Complex Event Processor for real-time analytics and some other components for batch processing and predictive analytics.

WSO2 Smart Analytics


Then what is the future of WSO2 CEP?
Still, WSO2 CEP is available on their site but users are recommended to migrate to DAS. Since all the WSO2 products are developed using the same framework called Carbon and also due to their similar aesthetic design across all the platforms, the transition from CEP to DAS should be easy sledding even for the beginners. Let me repeat, DAS is a creamed version of CEP which provides all the features of CEP as it was in CEP and some additional features for batch processing. This short post is to announce that the existing articles about WSO2 CEP are not going to be outdated or useless due to this transition. You can simply download and extract the DAS pack and test all the features I’ve discussed so far using DAS.

Is WSO2 CEP Dead? No! Here’s Why…


If you still have confusion, just go and download the DAS pack from the official site. Extract the pack and start the server using the following command:
./bin/wso2server.sh
Go to the admin dashboard and you will see all the features of WSO2 CEP except the color of the top banner ;-)
Therefore, my upcoming articles about complex event processing will use WSO2 DAS not CEP.
Read More

Apache Thrift Client for WSO2 CEP

In the series of WSO2 CEP tutorials, this article explains how to create Apache Thrift publisher and receiver for a CEP server in Python. Even though this is javahelps.com, I use Python since publisher and receiver in Java are already available in WSO2 CEP samples. 
One of the major advantages of Apache Thrift is the support for various platforms. Therefore this tutorial can be simply adapted to any other languages with less effort.

WSO2 CEP supports various protocols as a server to send and receive events. However using TCP protocol is recommended when the performance is a critical requirement. To achieve the security without performance trade-off, an SSL connection is used to authenticate the connection and the actuals events are be sent to the interested parties through unencrypted TCP connections.

Apache Thrift Client for WSO2 CEP

Requirements:
  • Apache Thrift
  • Python 2
  • Java Development Kit 1.7 or Latest
  • WSO2 Complex Event Processor 4.2.0

Setting up the environment
Step 1:
If you don't have Apache Thrift compiler, install it using the following command. Windows and Mac users, please find the binary files here.
sudo apt install thrift-compiler

Install the Python thrift library using the following command. Windows and Mac users must install this library using pip or manually include the library into the project.
sudo apt install python-thrift

Step 2:
Download and extract the WSO2 CEP pack. Start the server using the following command.
sh <CEP_HOME>/bin/wso2server.sh

Step 3:
Create a new stream using the following properties.
Event Stream Name com.javahelps.stream.Temperature
Payload data attributes
  • location STRING
  • temp INT

Step 4:
Create a publisher for testing purposes using the following configurations:
Event Publisher Name com.javahelps.publisher.logger.Temperature
Event Source com.javahelps.stream.Temperature
Output Event Adapter Type logger
Message Format text
Apache Thrift Client for WSO2 CEP
Now we have the required platform and let's jump into coding.

Step 1:
Create a new workspace directory wherever you like. Create a child directory wso2-thrift inside it and download the following files into the wso2-thrift directory from WSO2 GitHub repository.
  • Data.thrift
  • Exception.thrift
  • ThriftEventTransmissionService.thrift
  • ThriftSecureEventTransmissionService.thrift
Step 2:
Open the terminal inside the parent directory and execute the following commands to generate Python source files.
thrift -r --gen py wso2-thrift/Data.thrift
thrift -r --gen py wso2-thrift/Exception.thrift
thrift -r --gen py wso2-thrift/ThriftEventTransmissionService.thrift
thrift -r --gen py wso2-thrift/ThriftSecureEventTransmissionService.thrift
After executing these commands, you will get a gen-py directory which contains the generated source files.

Step 3:
The thrift receiver will run as a server and which requires an SSL certificate. Create the certificate using the following command. You need to answer the questions asked by openssl during this process.
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt

For example:
Country Name (2 letter code) [AU] CA
State or Province Name (full name) [Some-State] Ontario
Locality Name (eg, city) [] London
Organization Name (eg, company) [Internet Widgits Pty Ltd] Western University
Organizational Unit Name (eg, section) []: Faculty Of Engineering
Common Name (e.g. server FQDN or YOUR name) [] Gobinath
Email Address [] admin@javahelps.com

Step 5:
Once you have created the server.crt and server.key files, combine them into server.pem file using the following command.
cat server.crt server.key > server.pem

Step 6:
To register our SSL server as a reliable client to the CEP, enter the following command and restart the CEP.
keytool -importcert -file server.crt -keystore <CEP_HOME>/repository/resources/security/client-truststore.jks -alias "PythonThriftClient"

In the above command, the alias  PythonThriftClient can be anything and the <CEP_HOME> must be replaced by the actual home directory of WSO2 CEP.

When prompted, enter the keystore password 'wso2carbon' and confirm to trust the certificate.

Step 7:
Create a new file ServerHandler.py with the following code:
#!/usr/bin/env python


class ServerHandler(object):
    """
        ServerHandler contains the functions to serve the requests from WSO2 CEP.
    """
    def __init__(self):
        pass

    """
        Receive the username and password, verify it and return a unique session id.
    """
    def connect(self, uname, password):
        print 'Connect ' + uname + ':' + password
        return '123456'  # A random session id

    """
        Destroy the session.
    """
    def disconnect(self, sessionId):
        print 'Disconnect the session ' + str(sessionId)

    def defineStream(self, sessionId, streamDefinition):
        return ''

    def findStreamId(self, sessionId, streamName, streamVersion):
        return ''

    """
        Receive the event and process it.
    """
    def publish(self, eventBundle):
        print 'Received a new event: ' + str(eventBundle) + '\n'

    def deleteStreamById(self, sessionId, streamId):
        pass

    def deleteStreamByNameVersion(self, sessionId, streamName, streamVersion):
        pass

Step 8:
Create a new file TCPServer.py with the following code:
#!/usr/bin/env python

import sys
sys.path.append('gen-py')
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer
from ServerHandler import ServerHandler
from ThriftEventTransmissionService import ThriftEventTransmissionService

SERVER_ADDRESS = 'localhost'
SERVER_PORT = 8888

handler = ServerHandler()
processor = ThriftEventTransmissionService.Processor(handler)
transport = TSocket.TServerSocket(SERVER_ADDRESS, SERVER_PORT)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()

server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

print 'Starting TCP Server at ' + SERVER_ADDRESS + ':' + str(SERVER_PORT)

server.serve()

Note the server port is 8888

Step 9:
Create a new file SSLServer.py with the following code:
#!/usr/bin/env python

import sys
import os
sys.path.append('gen-py')
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from thrift.server import TServer
from thrift.transport import TSSLSocket
from ServerHandler import ServerHandler
from ThriftSecureEventTransmissionService import ThriftSecureEventTransmissionService


SERVER_ADDRESS = 'localhost'
SERVER_PORT = 8988
CERTIFICATE_FILE = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'server.pem')

handler = ServerHandler()
processor = ThriftSecureEventTransmissionService.Processor(handler)
transport = TSSLSocket.TSSLServerSocket(SERVER_ADDRESS, SERVER_PORT, certfile=CERTIFICATE_FILE)
tfactory = TTransport.TBufferedTransportFactory()
pfactory = TBinaryProtocol.TBinaryProtocolFactory()

server = TServer.TSimpleServer(processor, transport, tfactory, pfactory)

print 'Starting SSL Server at ' + SERVER_ADDRESS + ':' + str(SERVER_PORT)

server.serve()

Note the server port is 8988

Step 10:
Create a new file Publisher.py with the following code:
#!/usr/bin/env python

import sys
import time
sys.path.append('gen-py')

from ThriftSecureEventTransmissionService import ThriftSecureEventTransmissionService
from ThriftEventTransmissionService import ThriftEventTransmissionService
from ThriftEventTransmissionService.ttypes import *

from thrift import Thrift
from thrift.transport import TSSLSocket
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol


class Publisher:

    """
        Create SSL and TCP sockets along with buffer and binary protocol.
    """
    def __init__(self, ip, ssl_port, tcp_port):
        # Make SSL socket
        self.ssl_socket = TSSLSocket.TSSLSocket(ip, ssl_port, False)
        self.ssl_transport = TTransport.TBufferedTransport(self.ssl_socket)
        self.ssl_protocol = TBinaryProtocol.TBinaryProtocol(self.ssl_transport)

        # Make TCP socket
        self.tcp_socket = TSocket.TSocket(ip, tcp_port)
        self.tcp_transport = TTransport.TBufferedTransport(self.tcp_socket)
        self.tcp_protocol = TBinaryProtocol.TBinaryProtocol(self.tcp_transport)

    def connect(self, username, password):
        # Create a client to use the protocol encoder
        self.ssl_client = ThriftSecureEventTransmissionService.Client(self.ssl_protocol)
        self.tcp_client = ThriftEventTransmissionService.Client(self.tcp_protocol)

        # Make connection
        self.ssl_socket.open()
        # self.transport.open()
        self.sessionId = self.ssl_client.connect(username, password)

        self.tcp_socket.open()

    def defineStream(self, streamDef):
        # Create Stream Definition
        return self.tcp_client.defineStream(self.sessionId, streamDef)

    def publish(self, streamId, *attributes):
        # Build thrift event bundle
        event = EventBundle()
        event.setSessionId(self.sessionId)
        event.setEventNum(1)
        event.addLongAttribute(time.time() * 1000)
        event.addStringAttribute(streamId)
        for attr in attributes:
            if isinstance(attr, int):
                event.addIntAttribute(attr)
            elif isinstance(attr, basestring):
                event.addStringAttribute(attr)
            elif isinstance(attr, long):
                event.addLongAttribute(attr)
            elif isinstance(attr, float):
                event.addDoubleAttribute(attr)
            elif isinstance(attr, bool):
                event.addBoolAttribute(attr)
            else:
                event.setArbitraryDataMapMap(attr)

        # Publish
        self.tcp_client.publish(event.getEventBundle())

    def disconnect(self):
        # Disconnect
        self.ssl_client.disconnect(self.sessionId)
        self.ssl_transport.close()
        self.ssl_socket.close()

        self.tcp_transport.close()
        self.tcp_socket.close()


class EventBundle:
    __sessionId = ""
    __eventNum = 0
    __intAttributeList = []
    __longAttributeList = []
    __doubleAttributeList = []
    __boolAttributeList = []
    __stringAttributeList = []
    __arbitraryDataMapMap = None

    def setSessionId(self, sessionId):
        self.__sessionId = sessionId

    def setEventNum(self, num):
        self.__eventNum = num

    def addIntAttribute(self, attr):
        self.__intAttributeList.append(attr)

    def addLongAttribute(self, attr):
        self.__longAttributeList.append(attr)

    def addDoubleAttribute(self, attr):
        self.__doubleAttributeList.append(attr)

    def addBoolAttribute(self, attr):
        self.__boolAttributeList.append(attr)

    def addStringAttribute(self, attr):
        self.__stringAttributeList.append(attr)

    def setArbitraryDataMapMap(self, attr):
        self.__arbitraryDataMapMap = attr

    def getEventBundle(self):
        return Data.ttypes.ThriftEventBundle(self.__sessionId, self.__eventNum, self.__intAttributeList, self.__longAttributeList, self.__doubleAttributeList, self.__boolAttributeList, self.__stringAttributeList, self.__arbitraryDataMapMap)

Step 11:
Create a new file PublisherClient.py with the following code:
#!/usr/bin/env python

from Publisher import Publisher

CEP_SERVER_ADDRESS = '192.168.122.1' # IP address of the server. You can find it at the end of the CEP console
SSL_PORT = 7711  # Thrift SSL port of the server
TCP_PORT = 7611     # Thrift TCP port of the server
USERNAME = 'admin' # Username
PASSWORD = 'admin'  # Passowrd

publisher = Publisher(CEP_SERVER_ADDRESS, SSL_PORT, TCP_PORT)

# Connect to server with username and password
publisher.connect(USERNAME, PASSWORD)

# Publish an event to the Temperature stream
publisher.publish('com.javahelps.stream.Temperature:1.0.0', 'Kitchen', 56)

# Disconnect
publisher.disconnect()
Change the CEP_SERVER_ADDRESS according to your system. The SSL_PORT and TCP_PORT of the CEP must be as given default values. If they differ, we will update them in Step 15.

See the complete directory structure below:
.
├── gen-py
│   ├── Data
│   │   ├── constants.py
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── ttypes.py
│   │   └── ttypes.pyc
│   ├── Exception
│   │   ├── constants.py
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── ttypes.py
│   │   └── ttypes.pyc
│   ├── __init__.py
│   ├── ThriftEventTransmissionService
│   │   ├── constants.py
│   │   ├── __init__.py
│   │   ├── __init__.pyc
│   │   ├── ThriftEventTransmissionService.py
│   │   ├── ThriftEventTransmissionService.pyc
│   │   ├── ThriftEventTransmissionService-remote
│   │   ├── ttypes.py
│   │   └── ttypes.pyc
│   └── ThriftSecureEventTransmissionService
│       ├── constants.py
│       ├── __init__.py
│       ├── __init__.pyc
│       ├── ThriftSecureEventTransmissionService.py
│       ├── ThriftSecureEventTransmissionService.pyc
│       ├── ThriftSecureEventTransmissionService-remote
│       ├── ttypes.py
│       └── ttypes.pyc
├── PublisherClient.py
├── Publisher.py
├── server.crt
├── ServerHandler.py
├── server.key
├── server.pem
├── SSLServer.py
├── TCPServer.py
└── wso2-thrift
    ├── Data.thrift
    ├── Exception.thrift
    ├── ThriftEventTransmissionService.thrift
    └── ThriftSecureEventTransmissionService.thrift

Step 12:
Go to the CEP dashboard and create a new WSO2 event receiver. As shown below.
Event Receiver Name com.javahelps.receiver.Temperature
Input Event Adapter Type wso2event
Event Stream com.javahelps.stream.Temperature:1.0.0
Message Format wso2event
Apache Thrift Client for WSO2 CEP

Compare the ports available for Thrift protocol with the SSL_PORT and TCP_PORT in  PublisherClient.py . If there are any differences, update the  SSL_PORT and TCP_PORT values.

Step 13:
Add another publisher to publish the events to our Python servers. Use the following configuration.
Event Publisher Name com.javahelps.publisher.thrift.Temperature
Event Source com.javahelps.stream.Temperature
Output Event Adapter Type wso2event
Receiver URL tcp://localhost:8888
Authenticator URL ssl://localhost:8988
Username admin (Can be anything. Our authenticator accepts any username)
Password admin (Can be anything. Our authenticator accepts any password)
Protocol thrift
Publishing Mode non-blocking
Publishing Timeout 0
Message Format wso2event
Apache Thrift Client for WSO2 CEP

Step 14:
Start the TCPServer.py and SSLSErver.py using the following commands.
python TCPServer.py
python SSLSErver.py

Step 15:
Send an event to the WSO2 CEP server from PublisherClient using the following command and see the results in terminal.
python  PublisherClient.py

Apache Thrift Client for WSO2 CEP

Now we have successfully created a Thrift event publisher and receiver to send and receive events to and from WSO2 Complex Event Processor. To implement such a client in a different language, you need to change the Python related steps only. The Python clients along with the CEP artifacts are available at the following GitHub repository.

Find the project @ GitHub.
Read More
Siddhi 4.0.0 Early Access

Siddhi 4.0.0 Early Access

Siddhi 4.0.0 is being developed using Java 8 with major core level changes and features. One of the fundamental change to note is that some features of Siddhi have been moved as external extensions to Siddhi and WSO2 Complex Event Processor. This tutorial shows you, how to migrate the complex event project developed using Siddhi 3.x to Siddhi 4.x. Take the sample project developed in "Complex Event Processing - An Introduction" article which was developed using Siddhi 3.1.0.

Warning: Still, Siddhi 4.0.0 is not production ready and it is not recommended to use it until officially released by WSO2.

Required Tools:
  • Git
  • Apache Maven
  • Java 1.8
  • Eclipse or IntelliJ IDEA

Since Siddhi 4.0.0 is not officially released yet, we are going to compile the Siddhi 4.0.0-SNAPSHOT version manually.

Step 1:
Clone the Siddhi project using the following command
git clone https://github.com/wso2/siddhi.git

Step 2:
Set the following environment variable to avoid the Maven OutOfMemoryError.
MAVEN_OPTS="-Xms768m -Xmx3072m -XX:MaxPermSize=1200m"

Step 3:
Open the terminal in the siddhi directory and execute the following command to build Siddhi.
mvn clean install

You can reduce the time to build by skipping the tests using the following command
mvn clean install -Dmaven.test.skip=true

The external unique time batch window used in the "Complex Event Processing - An Introduction" is no more available in the Siddhi core. Most of the features are moved to a separate GitHub account known as "wso2-extensions". The external unique time batch window is available under the siddhi-execution-unique. Follow the steps to clone and build the unique extension.

Step 1:
Clone the siddhi-execution-unique project using the following command.
git clone https://github.com/wso2-extensions/siddhi-execution-unique.git

Step 2:
Open the terminal in the siddhi-execution-unique directory and execute the following command to build Siddhi.
mvn clean install

You can reduce the time to build by skipping the tests using the following command
mvn clean install -Dmaven.test.skip=true

Now clone the siddhidemo project which is from the Complex Event Processing - An Introduction" article or create a new project similar to that as suggested in that article.

Step 1:
Create a new Maven project in Eclipse using the following information.
Group Id: com.javahelps
Artifact Id: siddhimigrationdemo

Step 2:
Add the following dependencies to the pom.xml. Compared to Siddhi 3.x project, siddhi-execution-unique is required in addition.

Note: Check the pom file of Siddhi and siddhi-execution-unique projects and use the same versions in the following properties tag.
<properties>
    <siddhi.version>4.0.0-SNAPSHOT</siddhi.version>
    <siddhi.execution.unique.version>3.1.3-SNAPSHOT</siddhi.execution.unique.version>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
</properties>

<dependencies>
    <!--Siddhi -->
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-query-api</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-core</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-query-compiler</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
    <dependency>
        <groupId>org.wso2.extension.siddhi.execution.unique</groupId>
        <artifactId>siddhi-execution-unique</artifactId>
        <version>${siddhi.execution.unique.version}</version>
    </dependency>
</dependencies>

Step 3:
Create a new package com.javahelps.siddhimigrationdemo.

Step 4:
Create the Main.java as shown below. If you compare this class with the one used in Siddhi 3.x project, the way to use external time batch window is different. In Siddhi 3.x, the external time batch window is referred as #window.uniqueExternalTimeBatch but based on Siddhi 4.0.0 and the unique extension, it must be referred as #window.unique:externalTimeBatch.
package com.javahelps.siddhimigrationdemo;

import java.util.Random;

import org.wso2.siddhi.core.ExecutionPlanRuntime;
import org.wso2.siddhi.core.SiddhiManager;
import org.wso2.siddhi.core.event.Event;
import org.wso2.siddhi.core.stream.input.InputHandler;
import org.wso2.siddhi.core.stream.output.StreamCallback;

public class Main {
    //    SiddhiDe
    private static final String[] CUSTOMERS = {"Alice", "Barby", "Carol", "Diana"};
    private static final String[] ITEMS = {"Cocoa-Butter Lotion", "Purse-XL", "Purse-L", "Beer", "Biscuit",
            "Chocolate", "ZMA"};
    private static final Random RANDOM = new Random();

    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();

        String streams = "define stream purchaseStream (customerName string, item string, timestamp long); ";
        String query = "partition with (customerName of purchaseStream) " +
                "begin " +
                "from purchaseStream[item == 'Cocoa-Butter Lotion' OR item == 'Purse-XL' OR item == 'ZMA']#window" +
                ".unique:externalTimeBatch(item, timestamp, 500 milliseconds) " +
                "select customerName, convert(count(item), 'double') / 3.0 * 100.0 as noOfPurchases insert into " +
                "possiblePregnant; " +
                "end ";

        // Create ExecutionPlanRuntime using stream definition and query
        ExecutionPlanRuntime executionPlanRuntime = siddhiManager.createExecutionPlanRuntime(streams + query);

        try {
            // Receive the output events
            executionPlanRuntime.addCallback("possiblePregnant", new StreamCallback() {
                @Override
                public void receive(Event[] events) {
                    String output = String.format("\t\t\t%s is pregnant with %.2f%% confidence.", events[0].getData(0),
                            events[0].getData(1));
                    System.out.println(output);
                }
            });

            // Send input events
            InputHandler purchaseStream = executionPlanRuntime.getInputHandler("purchaseStream");
            executionPlanRuntime.start();
            for (int i = 0; i < 1000; i++) {
                Object[] event = generateEvent();
                purchaseStream.send(event);
                Thread.sleep(10); // Delay for 10 milliseconds
            }
        } finally {
            executionPlanRuntime.shutdown();
        }
    }

    private static Object[] generateEvent() {
        String name = CUSTOMERS[RANDOM.nextInt(CUSTOMERS.length)];
        String item = ITEMS[RANDOM.nextInt(ITEMS.length)];
        long time = System.currentTimeMillis(); // Current time

        System.out.println(name + " buys " + item);
        Object[] event = new Object[]{name, item, time};
        return event;
    }

}

Save all the changes and run the Main.java. You will see the same results as the Siddhi 3.x project.

Find the project @ GitHub.
Read More

JPA - Hello, World! using Hibernate 5

The article JPA - Hello, World! using Hibernate explains how to use Hibernate v4.x. This tutorial introduces using JPA with the latest Hibernate v5.2.3. Same as the previous article, it also targets beginners who are new to Hibernate and JPA.

You need Java 1.8, Eclipse for Java EE developers and MySQL server in your system in-order to try this tutorial.

Step 1:
Create a database “javahelps” and a table “student” in MySQL.
CREATE DATABASE IF NOT EXISTS javahelps;

CREATE  TABLE javahelps.student ( 
student_id INT NOT NULL ,
student_name VARCHAR(45) NOT NULL ,
student_age INT NOT NULL ,
PRIMARY KEY (student_id) );

Read More

WSO2 CEP - Output Mapping Using Registry Resource

Publishing the output is an important requirement of CEP. WSO2 CEP allows to convert an event to TEXT, XML or JSON, which is known as output mapping . This article explains how a registry resource can be used for custom event mapping in WSO2  CEP 4.2.0.


Step 1:
Start the WSO2 CEP and login to the management console.

Step 2:
Navigate to Home → Manage → Events → Streams → Add Event Stream.

Step 3:
Define an event stream as given below and click on ‘Add Event Stream’ button.
Name: sensorstream
Version: 1.0.0
Meta Data Attributes: sensorId, sensorName and language
Correlation Data Attributes: longitude and latitude
Payload Data Attributes: sensorValue


Step 4:
Navigate to Home → Registry → Browse.

Step 5:
Expand the /_system/config tree view and click on the config folder.

Step 6:
Click on the Add Collection link and add a new collection named template.


Step 7:
Same as Step 6, create another sub folder en inside the template folder.

Step 8:
Click on the ‘Add Resource’ button and select ‘Create Text Content’ as the method. This means that we are going to provide a content that is readable for human.
Provide the name as message and leave the Media type as text/plain. Suppose you want to create an XML or JSON resource, change the media type to application/xml or application/json respectively.

Step 9:
Add the content that matches your media type and click the Add button.


The message used in this screenshot is given below:
Welcome
Sensor {{meta_sensorName}} reports {{sensorValue}} from {{correlation_longitude}}:{{correlation_latitude}}

Step 10:
Navigate to the Home → Manage → Event → Publishers and click on the Add Event Publisher button.

Step 11:
Provide samplelogger as the publisher name, sensorstream:1.0.0 as the event source, logger as the output event adapter type and text as the message format. If your registry resource is XML or JSON, change the message format to your media type.

Then expand the Advanced configuration and select ‘Pick from registry’.
Click on the ‘Configuration Registry’ button, browse and select the message resource.


Leave the Cache Timeout with the default value.

Compare your configuration with the following screenshot.


Finally click the Add button.

Step 12:
Now move to the  Home → Tools → Event Simulator and select the sensorstream:1.0.0 as the events stream name.
Fill the form as shown below and click the send button.
sensorId: 10
sensorName: Temperature
language: en
longitude: 79.861256
latitude: 6.927131
sensorValue: 23.0


Once you clicked the Send button, you should get the following output in the terminal of CEP.



Parametrized Registry Path
WSO2 CEP allows you to parameterize the registry path as well. For example, assume that we want to select a registry resource depending on the meta_language. For this purpose, we need another registry resource.

As you did in Step 4 – 9, cerate a new folder named fr in /_system/config/template and a resource named message in /_system/config/template/fr folder.


The message used in this screenshot is given below:
Bienvenue
Sensor {{meta_sensorName}} reports {{sensorValue}} from {{correlation_longitude}}:{{correlation_latitude}}

Now the registry should contain both en/message and fr/message.


Step 13:
Delete the existing samplelogger publisher and create a new one using the same name as shown in Step 11. But this time, provide the following path as the registry path.
conf:/template/{{meta_language}}/message
Compare this path with the path in Step 11. Instead of hard coding the language folder name en or fr, the attribute meta_language is used. At the runtime, the meta_language will be replaced by the actual language name and the resource at that path will be used by the publisher.


Step 14:
Repeat the Step 12 with languages en, fr and something other than English and French.
In the terminal, CEP will print the message in English and French according to the input and thrown an exception if the resource for the given language does not exists.


If the registry does not contain a resource at the run time evaluated path, an exception will be thrown.

For more details about output mapping, visit to the official document: Output Mapping Types
Read More

Never Duplicate A Window Again - WSO2 Siddhi Event Window

A new feature known as Event Window is introduced in WSO2 Siddhi 3.1.1 version, which is quite similar to the named window of Esper CEP in some aspects. This article presents the application and the architecture of Event Window using a simple example. According to Siddhi version 3.1.0, a window can be defined on a stream inside a query and the output can be used in the same query itself. For example, consider a scenario where WSO2 CEP is used to analyze the sensor reading of a smart home which has multiple sensors in number of rooms. All the sensors are sending their reading to an input stream named SensorStream and there are different analytics components interested in various statistic information.

If there are two analytic components named A and B which are interested in the maximum reading of each sensors in every room in last 5 seconds and the average sensor reading of each type of sensors in the smart home in last 5 seconds respectively, it can be achieved using a query provided below.
define stream SensorStream (name string, value float, roomNo int, deviceID string);

@info(name = 'query0')
from SensorStream#window.timeBatch(1 second)
select name, max(value) as maxValue, roomNo
group by name, roomNo
insert into MaxSensorReadingPerRoomStream;

@info(name = 'query1')
from SensorStream#window.timeBatch(1 second)
select name, max(value) as maxValue, roomNo
group by name, roomNo
insert into AverageSensorReadingPerBuildingStream;
In this definition, the events arrived in last second will be stored in two windows which are identical  to each other. The output is determined only by the selection and grouping. If the time interval is large enough, with thousands of sensors used in a smart house, several megabytes of memory will be wasted by the duplicate events. Also the window defined in a query cannot be reused for any other purpose even inside the same query. To overcome these problems, Event Window is introduced as a global window which can be accessed from any queries any number of times.

The Siddhi queries and the architecture discussed in this article are presented according to the Siddhi version 3.1.1.

Read More

Deploy and Upgrade Android Database From External Directory

After seeing the huge response for the Import and Use External Database in Android article, I have realized the importance of deploying Android database from external sources. The library used in the above article allows you to import database only from the assets directory. However, recently one of my readers, asked for a way to import and upgrade the database from SD card. As a solution for his use case, I have developed a new library named "externalsqliteimporter" which allows you to import database either from assets directory or from SD card. This article explains the application of this Android library using a sample application.


The ExternalSQLiteImporter library allows you to build your SQLite database on your desktop computer, and to import and use it in your Android application. This library has two separate ways to maintain your database.

This library is still under development. Deploying and upgrading  the database from an external directory is not secure as it is publicly available for third party applications as well. Use that feature with caution. 

Read More

Complex Event Processing - An Introduction

Today almost all the big brothers in the software industry are behind big data and data analytics. Not only the large companies, even small scale companies need data processing in order to track and lead their business. Complex Event Processing(CEP) is one of the techniques being used to analyse streams of events for interested events or patterns. This article explains the big picture of complex event processing in a nutshell using a simple example.


As the name suggests Complex Event Processing is mainly about processing events using some predefined rules. Mainly they are used to derive complex events by aggrgating and processing stream of simple events. For example, identifying that you are reading this blog right now is an easy task for your computer since the browser knows the site which is currently being opened on your computer. However, some events cannot be detected as simple as the above example. For example, how could a supermarket send coupons closed to a teen girl’s delivery date while even her father didn’t know that she was pregnant? (Read the incredible story here: ‘How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did’).

Let’s say our problem is identifying the pregnant customers of a supermarket so that the supermarket can send  coupons for baby clothes and cribs. Assume that according to experts report, if a customer bought cocoa-butter lotion, a purse large enough to double as a diaper bag and ZMA supplements within 7 days, the customer is pregnant (or he/she has a pregnant relation). Using this domain knowledge, we can define the rule to identify the pregnant customers. However, in a supermarket with thousands of customers per day, it is hard to search for such patterns manually using a rule.

Complex Event Processor is a tool to process stream of events automatically as they arrive. Compared to database management systems, database engines store the data and pass the query to process on those data. In contrast, CEPs store the query and pass the data against it and extract the necessary information. There are several CEPs available and this article uses WSO2 CEP which is an open source Java based complex event processor.

WSO2 CEP is recommended to be used as a cluster of nodes for an enterprise level requirement. However, in this article, we use the core of WSO2 CEP known as Siddhi which can be used as a Java library.

Step 1:
Create a new Maven project in Eclipse using the following information.
Group Id: com.javahelps
Artifact Id: siddhidemo

Step 2:
Open the pom.xml and add the following repository. This repository is required to download the latest Siddhi library.
<repositories>
    <repository>
        <id>wso2.releases</id>
        <name>WSO2 internal Repository</name>
        <url>http://maven.wso2.org/nexus/content/repositories/releases/</url>
        <releases>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>
            <checksumPolicy>ignore</checksumPolicy>
        </releases>
    </repository>
</repositories>

Step 3:
Add the following dependencies. (Java version is optional)
<properties>
    <siddhi.version>3.1.0</siddhi.version>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
</properties>

<dependencies>
    <!--Siddhi -->
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-query-api</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-core</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
    <dependency>
        <groupId>org.wso2.siddhi</groupId>
        <artifactId>siddhi-query-compiler</artifactId>
        <version>${siddhi.version}</version>
    </dependency>
</dependencies>

After modification, the pom.xml must look like this:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.javahelps</groupId>
    <artifactId>siddhidemo</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <repositories>
        <repository>
            <id>wso2.releases</id>
            <name>WSO2 internal Repository</name>
            <url>http://maven.wso2.org/nexus/content/repositories/releases/</url>
            <releases>
                <enabled>true</enabled>
                <updatePolicy>daily</updatePolicy>
                <checksumPolicy>ignore</checksumPolicy>
            </releases>
        </repository>
    </repositories>

    <properties>
        <siddhi.version>3.1.0</siddhi.version>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>

    <dependencies>
        <!--Siddhi -->
        <dependency>
            <groupId>org.wso2.siddhi</groupId>
            <artifactId>siddhi-query-api</artifactId>
            <version>${siddhi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.wso2.siddhi</groupId>
            <artifactId>siddhi-core</artifactId>
            <version>${siddhi.version}</version>
        </dependency>
        <dependency>
            <groupId>org.wso2.siddhi</groupId>
            <artifactId>siddhi-query-compiler</artifactId>
            <version>${siddhi.version}</version>
        </dependency>
    </dependencies>
</project>

Step 4:
Create a new package com.javahelps.siddhidemo in src/main/java folder.

Step 5:
Create a new class named Main.java with the main method.

Step 6:
Create a SiddhiManager instance which is used to create an execution plan.
package com.javahelps.siddhidemo;

import org.wso2.siddhi.core.SiddhiManager;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();
    }
}

Step 7:
Define a stream as shown below. Here stream is used to define the properties of the input event.
package com.javahelps.siddhidemo;

import org.wso2.siddhi.core.SiddhiManager;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();

        String streams = "define stream purchaseStream (customerName string, item string, timestamp long); ";
    }
}
The above definition says, all the events passed to the purchaseStream have the properties customerName and item which are String and timestamp of purchase in long.

Step 8:
The most important step is creating the query.
Here is my logic:
  1. Get all the streams and partition them based on the customer name because there can be more customers and we need to process their data individually.
  2. From every partitions, filter the purchases of interested items.
  3. Again break the events into subsets where each set contains only purchases happened in 7 days interval. For example, if Alice purchased Cocoa-Butter Lotion on Monday, Biscuit on Saturday and Beer on next Tuesday, the  Cocoa-Butter Lotion and Biscuit purchases must be in one subset and Beer must be in another subset.
  4. From each subset of events, take the count of unique items. Count of unique items can range from 1 to 3 since we have already filtered the stream to allow purchases of three items only.
  5. Those who have purchased all three products are pregnant with 100% confidence. Those who have  bought only 2 items are pregnant with 66.67% confidence. Those who have purchased only one product are pregnant with 33.33% confidence.
The Siddhi query for above logic is given below:
partition with (customerName of purchaseStream)
begin
    from purchaseStream[item == 'Cocoa-Butter Lotion' OR item == 'Purse-XL' OR item == 'ZMA']#window.uniqueExternalTimeBatch(item, timestamp, 500 milliseconds)
    select customerName, convert(count(item), 'double') / 3.0 * 100.0 as noOfPurchases insert into possiblePregnant;
end;

1. The input stream is partitioned by customerName.

2. The partitioned stream is filtered to allow only Cocoa-Butter Lotion, Purse-XL and ZMA.

3. Siddhi offers uniqueExternalTimeBatch which can be used to get unique time framed subset of events. The interval of 500 milliseconds is used instead of 7 days for quick response.

4. Select the customerName and count of items from each subset and insert the possible pregnant customers with the level of confidence into the output stream.

Step 9:
Add the stream definition and the query to the Siddhi manager and create the ExecutionPlanRuntime.
package com.javahelps.siddhidemo;

import org.wso2.siddhi.core.ExecutionPlanRuntime;
import org.wso2.siddhi.core.SiddhiManager;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();

        String streams = "define stream purchaseStream (customerName string, item string, timestamp long); ";
        String query = "partition with (customerName of purchaseStream) " +
                        "begin " +
                        "from purchaseStream[item == 'Cocoa-Butter Lotion' OR item == 'Purse-XL' OR item == 'ZMA']#window.uniqueExternalTimeBatch(item, timestamp, 500 milliseconds) " +
                        "select customerName, convert(count(item), 'double') / 3.0 * 100.0 as noOfPurchases insert into possiblePregnant; " +
                        "end ";

        // Create ExecutionPlanRuntime using stream definition and query
        ExecutionPlanRuntime executionPlanRuntime = siddhiManager.createExecutionPlanRuntime(streams + query);
    }
}

Step 10:
Create two static arrays one for Customers and the other for Products and a static Random class and add a new method to generate input events.
package com.javahelps.siddhidemo;

import java.util.Random;

import org.wso2.siddhi.core.ExecutionPlanRuntime;
import org.wso2.siddhi.core.SiddhiManager;

public class Main {
    private static final String[] CUSTOMERS = { "Alice", "Barby", "Carol", "Diana" };
    private static final String[] ITEMS = { "Cocoa-Butter Lotion", "Purse-XL", "Purse-L", "Beer", "Biscuit",
            "Chocolate", "ZMA" };
    private static final Random RANDOM = new Random();

    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();

        String streams = "define stream purchaseStream (customerName string, item string, timestamp long); ";
        String query = "partition with (customerName of purchaseStream) " +
                        "begin " +
                        "from purchaseStream[item == 'Cocoa-Butter Lotion' OR item == 'Purse-XL' OR item == 'ZMA']#window.uniqueExternalTimeBatch(item, timestamp, 500 milliseconds) " +
                        "select customerName, convert(count(item), 'double') / 3.0 * 100.0 as noOfPurchases insert into possiblePregnant; " +
                        "end ";

        // Create ExecutionPlanRuntime using stream definition and query
        ExecutionPlanRuntime executionPlanRuntime = siddhiManager.createExecutionPlanRuntime(streams + query);
    }

    private static Object[] generateEvent() {
        String name = CUSTOMERS[RANDOM.nextInt(CUSTOMERS.length)];
        String item = ITEMS[RANDOM.nextInt(ITEMS.length)];
        long time = System.currentTimeMillis(); // Current time

        System.out.println(name + " buys " + item);
        Object[] event = new Object[] { name, item, time };
        return event;
    }
}

Step 11:
Add StreamCallback to receive the output events and InputHandler to send events.
package com.javahelps.siddhidemo;

import java.util.Random;

import org.wso2.siddhi.core.ExecutionPlanRuntime;
import org.wso2.siddhi.core.SiddhiManager;
import org.wso2.siddhi.core.event.Event;
import org.wso2.siddhi.core.stream.input.InputHandler;
import org.wso2.siddhi.core.stream.output.StreamCallback;

public class Main {
    private static final String[] CUSTOMERS = { "Alice", "Barby", "Carol", "Diana" };
    private static final String[] ITEMS = { "Cocoa-Butter Lotion", "Purse-XL", "Purse-L", "Beer", "Biscuit",
            "Chocolate", "ZMA" };
    private static final Random RANDOM = new Random();

    public static void main(String[] args) throws InterruptedException {
        SiddhiManager siddhiManager = new SiddhiManager();

        String streams = "define stream purchaseStream (customerName string, item string, timestamp long); ";
        String query = "partition with (customerName of purchaseStream) " +
                        "begin " +
                        "from purchaseStream[item == 'Cocoa-Butter Lotion' OR item == 'Purse-XL' OR item == 'ZMA']#window.uniqueExternalTimeBatch(item, timestamp, 500 milliseconds) " +
                        "select customerName, convert(count(item), 'double') / 3.0 * 100.0 as noOfPurchases insert into possiblePregnant; " +
                        "end ";

        // Create ExecutionPlanRuntime using stream definition and query
        ExecutionPlanRuntime executionPlanRuntime = siddhiManager.createExecutionPlanRuntime(streams + query);
       
        try {
            // Receive the output events
            executionPlanRuntime.addCallback("possiblePregnant", new StreamCallback() {
                @Override
                public void receive(Event[] events) {
                    String output = String.format("\t\t\t%s is pregnant with %.2f%% confidence.", events[0].getData(0),
                            events[0].getData(1));
                    System.out.println(output);
                }
            });

            // Send input events
            InputHandler purchaseStream = executionPlanRuntime.getInputHandler("purchaseStream");
            executionPlanRuntime.start();
            for (int i = 0; i < 1000; i++) {
                Object[] event = generateEvent();
                purchaseStream.send(event);
                Thread.sleep(10); // Delay for 10 milliseconds
            }
        } finally {
            executionPlanRuntime.shutdown();
        }
    }

    private static Object[] generateEvent() {
        String name = CUSTOMERS[RANDOM.nextInt(CUSTOMERS.length)];
        String item = ITEMS[RANDOM.nextInt(ITEMS.length)];
        long time = System.currentTimeMillis(); // Current time

        System.out.println(name + " buys " + item);
        Object[] event = new Object[] { name, item, time };
        return event;
    }

}

Step 12:
Save all the changes and run the application.

The above query is a sample implementation for the given scenario. There can be other ways to achieve the same output using CEP. For more details about Siddhi, visit to the following link: Siddhi Documentation. This example is a simple application of CEP to process stream of events, however CEP can be used in complex scenarios like weather prediction, the stock market trends, fraud detection in bank transaction, etc as well. If you have any questions or suggestions, please comment below.

Find the project @ Git Hub.
Read More

WSO2 Carbon UI Component - Hello World!

In this article, I will show you how to create a WSO2 Carbon UI front end for an existing web service. Since this is the part II of the last article “WSO2 Carbon Server Component - Hello World!”, the UI will be created for the web service developed in part I. However, the same steps can be followed to create a Carbon UI for any SOAP web services.

Prerequisite:

Read More

Goodbye Log4j

After seeing so many students in last four years, I have decided to write this article about the new loggers which are widely being used by the industry. Almost all the university students I have seen are familiar with Log4j 1.x (at least heard about it) but most of them even did not hear about SLF4J and Logback. The purpose of this article is introducing SLF4J and Logback and convincing you towards them. Before getting into the topic, be informed that Log4j 1.x is not being maintained after August 5, 2015 and Ceki Gülcü the developer of Log4j came up with the new tools SLF4J and Logback. Technically, Logback is an enhanced successor of Log4j and performs better than Log4j.

He did a good job, but we have to move forward.
Read More

Contact Form

Name

Email *

Message *