Spark 05: List Action Movies with Spark flatMap

Welcome to the fifth article in the series of Apache Spark tutorials. In this article, you will learn the application of flatMap transform operation. After the introduction to flatMap operation, a sample Spark application is developed to list all action movies from the MovieLens dataset.

In the previous articles, we have used the map transform operation which transforms an entity into another entity where the transformation is one-to-one. For example, suppose you have a String RDD named lines, applying lines.map(x => x.toUpperCase) operation creates a new String RDD with the same number of records but with uppercase string literals as shown below:

Read More

Install Ballerina on Linux

Ballerina is a new open source JVM based language specially designed for integration purposes by WSO2 the world's #1 open source integration vendor. In this article, you will see how to manually install Ballerina on Linux systems. Visit the official website and download the installer for your system. There is an installer for Windows, Mac, Debian-based Linux and Fedora-based Linux. I prefer to install Ballerina manually because it is universal for all Linux operating systems out there.

Read More

Spark 04: Key Value RDD and Average Movie Ratings

In the first article of this series: Spark 01: Movie Rating Counter, we created three RDDs (data, filteredData and ratingData) each contains a singular datatype. For example, data and filteredData were String RDDs and the ratingRDD was a Float RDD. However, it is common to use an RDD which can store complex datatypes especially Key-Value pairs depending on the requirement. In this article, we will use a Key-Value RDD to calculate the average rating of each movie in our MoviLens dataset.  Those who don't have the MovieLens dataset, please visit the Spark 01: Movie Rating Counter article to setup your environment.

As you already know, the ratings.csv file has the fields movieId and rating. A given movie may get different ratings from different users. To get the average ratings of each movie, we need to add all ratings of each movie individually and divide the sum by the number of ratings.

Read More

Spark 03: Understanding Resilient Distributed Dataset

You are not qualified as an Apache Spark developer until you know what is a Resilient Distributed Dataset (RDD). It is the fundamental technique to represent data in the Spark memory. There are advanced data representation techniques like DataFrame built on top of RDD. However, it is always better to start with the most basic dataset: RDD. RDD is nothing other than a data structure with some special properties or features.

We all know that Apache Spark is a distributed general-purpose cluster-computing framework. There are some common problems faced in a distributed environment including but not limited to:

1. Remote access of data is expensive
2. High chance of failure
3. Runtime errors are expensive and hard to track
4. Wasting computing power is way too expensive

RDD is designed to address the abovementioned problems. In the following section, you will see the properties of RDD and how it solves these problems.

Read More

Spark 02: Scala Cheat Sheet for Java Developers

This article introduces Scala to those Java developers who don't know Scala. I assume here that you already know Java (preferably Java 8) so that you can compare the features of Scala with Java. Please be informed that this article is not an end to end Scala tutorial. I am covering only the fundamentals of Scala which are used in my Apache Spark tutorials.

First of all, remember that Scala is a JVM based language which is running on top of your regular Java Virtual Machine. The whole purpose of Scala is providing a convenient functional programming language (at that time Java 8 wasn't there). However since Scala is built on top of Java, you can access Java libraries and API from your Scala code.

To play with Scala, please setup Scala on IntelliJ IDEA or install a command line Scala version in your system.

Read More

Spark 01: Movie Rating Counter

Apache Spark is a must to know framework for big data developers. This is an attempt to write a series of articles on Apache Spark to train you from zero to hero. In this series of articles, I will use the latest Apache Spark release which is 2.4.0 as of 2019 January. In the first few articles, we will code and test Apache Spark on IntelliJ Idea. As you may already know, Apache Spark is developed using Scala and of course, there are APIs available for other languages including Java and Python. Still, Scala has preferred over other languages for its performance and compact code. Therefore, you need to prepare the environment first.

My articles will be based on Frank Kane's course on Udemy: Apache Spark 2 with Scala - Hands On with Big Data! I highly recommend his tutorial if you prefer for a video tutorial.

Setup the Environment

Apache Spark 2.4.0 depends on Scala 2.12 which in turn depends on Java 1.8. Scala is unlike Java not very version compatible language. Therefore, please take special care on choosing versions.

Step 1:

Install Oracle Java Development Kit 1.8 on your system. Linux users can follow this article: Install Oracle JDK 8 on Linux

Step 2:

Install the latest IntelliJ IDEA. Again, Linux users can follow my article: Install IntelliJ IDEA on Linux

Step 3:
Install Scala plugin in IntelliJ IDEA. Regardless of your operating system, you can follow the article Setup Scala on IntelliJ IDEA.

Read More

Android: Custom Font in Kotlin

Sometimes you may want to use custom font in your Android application for aesthetic requirement or to show a message in a different language. I have already written an article on how to use a Custom Font in Android four years ago. Since it has been a long time and a reader wondered if that code still works, I am writing this new article using Kotlin. However, the underlying technique hasn't change over the years and you can still use my previous article.

Step 1:

Create an Android application "Custom Font" with Kotlin support.

Read More

Access MySQL from Android through RESTful Web Service

Finally, I found some time to write about accessing MySQL or any database that you can think about from an Android application. Android offers a built-in database known as SQLite which is supposed to be used for local data storage. For example, if you want to store your recent transactions locally in the device, you have to go with SQLite. I have already written a few articles on how to use SQLite database in Android. However, enterprise mobile applications often depend on client/server databases such as MySQL, Oracle DB or MongoDB. Whatever the database, you have chosen, the way to access them from an Android application is pretty same. In this article, I explain how to access a MySQL database from an Android application written in Kotlin.

I will use Kotlin for all upcoming Android tutorials because it is the future of Android programming. Kotlin is a JVM based language and you can easily convert a Kotlin project into Java and vice versa.

An Android application is not supposed to directly access a database deployed in a server. I do not recommend implementing JDBC connections in an Android application due to the tight coupling introduced between the application and the database. The industrial best practice requires you to implement a web service between the database and the Android application. Having a web service layer reduces the complexity of the Android application and also reduces the dependency on database specific operations. Therefore, the problem of accessing a client/server database like MySQL from an Android application can be defined as the problem of consuming a web service hosted somewhere.

Having said that, as an Android developer, you don't need to care about what is behind the web service. It may be a MySQL database, MongoDB database, a Social Media network or even a Weather Network API. What you need is the API endpoints exposed by the web service. In this article, I only explain how to connect to a web service hosted on your local machine from an Android application. However, the sample web service used in this article provides API endpoints to CRUD operations on a MySQL database. Please follow the RESTful CRUD With Java and MySQL in Minutes article and develop the web service to access a MySQL database. In the following section, you will learn how to access the web service you have created for accessing a database.

Read More

Setup Scala on IntelliJ IDEA

Scala is a JVM based functional programming language widely being used in data analytics. You can install Scala on Ubuntu using sudo apt install scala. However, IDE support is inevitable these days. Therefore, this article explains how to use Scala in IntelliJ IDEA.

Step 1:

Install Oracle JDK 1.8. Please note that Scala strictly requires Java 1.8. You can follow my article Install Oracle JDK 8 on Linux to install JDK 1.8 on Linux.

Step 2:

Install the latest IntelliJ IDEA. You can install either the community version or the Ultimate version depending on the license. Again, I have an article explaining how to Install IntelliJ IDEA on Linux.

Read More

Install IntelliJ IDEA on Linux

This article shows you the way to manually install the latest version of IntelliJ IDEA on Linux. If you do not have Java in your system, follow this link and install the Java first.

Read More

Android: Simple Calculator in Kotlin

It has been a long time since I wrote the tutorial: Android: Simple Calculator. However, still, that tutorial attracts more readers to my blog. Therefore, I have decided to write a new article to develop the same application using Kotlin. The previous article on developing a Calculator application using Java is still valid and feel free to compare this article with the previous one.

Step 1:

Create a new Android application: Kotlin Calculator with an Empty activity. In the very first dialog, make sure that you have enabled the Kotlin support.

Read More

Determine if a String has all Unique Characters

Question: Implement an algorithm to determine if a string has all unique characters.

Example:

The String "Hello" has two "l"s. Therefore, it does not have all unique characters. On the other hand, if you consider the String "World", it is formed using all unique characters.

Break Down:
Let's write a class named TestUniqueness which has a method isUnique which can be called from the main method as given below:

public class TestUniqueness {

    public static void main(String[] args) {
        System.out.println(isUnique("hello"));
        System.out.println(isUnique("world"));
    }

    public static boolean isUnique(String text) {
        return false;
    }
}

Of course, you can write a fancy code to read user input and feed them to an isUnique method. However, to keep the code clean and simple, I am calling the static method isUnique and immediately print the response. The isUnique method returns false all the times. In the following section, we are going to implement the algorithm for the isUnique method.

Read More

Setup Apache Axis2 on Ubuntu

This tutorial helps you to setup Apache Axis2 on Ubuntu and to add it to Eclipse IDE for Java EE Developers.

Step 1:
Download the latest version of Apache Axis2 from this link.

Step 2:
Open the Terminal (Ctrl + Alt + T) and enter the following command to change the directory.

cd /opt/

Step 3:
Enter the command given below to extract the Axis2 from  the ~/Downloads directory. If your downloaded file is in any other directory and replace the last parameter by the actual file path.

sudo unzip ~/Downloads/axis2-1.7.8-bin.zip -d .

Step 4:
Rename the folder name to axis2.

sudo mv axis2-1.7.8/ axis2/

Step 5:
To deploy web services, you may need to add some files inside this folder. Therefore it is required to change the permission of this directory. Enter the following command to change the permission of axis2 folder.

sudo chmod -R 777 axis2/

Step 6:
Environment variables AXIS2_HOME and PATH have to be added to the system. Enter the following command in the terminal to open the /etc/environment.

sudo gedit /etc/environment

Step 7:
Add the following line at the end of the file.

AXIS2_HOME="/opt/axis2"

Add the /opt/axis2/bin to the PATH.

The /etc/environment before the modification.

The /etc/environment after the modification.

Step 8:
Reload the environment variables to the current terminal using this command.

source /etc/environment

Step 9:
Enter the following command to start the Axis2 server.

axis2server.sh

If you are using Java 9 or latest, you may get an error similar to:
"-Djava.endorsed.dirs=/opt/axis2/lib/endorsed:/usr/lib/jvm/jdk-10.0.2/jre/lib/endorsed:/usr/lib/jvm/jdk-10.0.2/lib/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit".
To fix this problem, open $AXIS2_HOME/bin/axis2Server.sh script and scroll down to the bottom. At the end of the script, you have a command like this:

java $JAVA_OPTS -classpath "$AXIS2_CLASSPATH" \
    -Djava.endorsed.dirs="$AXIS2_HOME/lib/endorsed":"$JAVA_HOME/jre/lib/endorsed":"$JAVA_HOME/lib/endorsed" \
    org.apache.axis2.transport.SimpleAxis2Server \
    -repo "$AXIS2_HOME"/repository -conf "$AXIS2_HOME"/conf/axis2.xml $*

Replace it with the following command. As you can notice, we simply remove the JVM arguments from the command.

java $JAVA_OPTS -classpath "$AXIS2_CLASSPATH" \
    org.apache.axis2.transport.SimpleAxis2Server \
    -repo "$AXIS2_HOME"/repository -conf "$AXIS2_HOME"/conf/axis2.xml $*

Save all the changes and start the server again.

Step 10:
Visit to the following URL.
http://localhost:8080/axis2/services/
If you get the Deployed services page, you have successfully setup the Apache Axis2 in your system.

Step 11:
To stop the Axis2, press Ctrl + C in the terminal which is currently running the axis2server.

Integrate Axis2 with Eclipse

If you do not have Eclipse IDE for Java EE Developers, follow this link and install the Eclipse.

Step 1:
Open the Eclipse.

Step 2:
Goto Windows → Preferences in the menu bar and select the Axis2 Preferences under the Web Services.

Step 3:
Click on the 'Browse' button and select the Axis2 home directory.

Step 4:
Click ‘Apply’ and ‘OK’

Now you are ready to use Apache Axis2 from Eclipse.

Read More