Run Apache NiFi Cluster in Docker

The last article on Apache NiFi: Run Apache NiFi in Docker was for those who want to start playing with Apache NiFi. Though it was a good start to play with NiFi, it is far from production deployment. This article introduces the second stage of deployment: a NiFi cluster running in Docker using Docker Compose.

Run Apache NiFi in Docker

To begin with, you must have Docker installed in your system and also install Docker Compose as we are going to use Docker Compose to setup the Apache NiFi cluster. 


Step 1:
Create a new file named docker-compose.yml with the following content:

version: "3"
services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: 'zookeeper:latest'
    ports:
      - 2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:latest
    ports:
      - 8080
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=xxxxxxxxxxxx

In the above configuration, change the value of the NIFI_SENSITIVE_PROPS_KEY with a strong 12 character password to encrypt the sensitive properties used in the NiFi configuration. Though the provided key works, it is not secured enough.


Step 2:
Run the following Docker command to run 3 instances of Apache NiFi. Note that NiFi instances can communicate between themselves with the help of Zookeeper and elect a node as the Cluster Coordinator. For more details about the Zero-Leader Clustering employed in Apache NiFi, check the official document.

docker-compose up --scale nifi=3
Run Apache NiFi Cluster in Docker

Step 3:
Run the following Docker command to list the currently running Docker containers.

$ docker container ls
CONTAINER ID   IMAGE                COMMAND                  CREATED             STATUS             PORTS                                                                                          NAMES
df73fe7dfad2   zookeeper:latest     "/docker-entrypoint.…"   About an hour ago   Up About an hour   2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:49180->2181/tcp, :::49180->2181/tcp                      zookeeper
e1ebfa5fe7e0   apache/nifi:latest   "../scripts/start.sh"    About an hour ago   Up About an hour   8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49177->8080/tcp, :::49177->8080/tcp                     http_nifi_2
455cb9f8ff1f   apache/nifi:latest   "../scripts/start.sh"    About an hour ago   Up About an hour   8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49178->8080/tcp, :::49178->8080/tcp                     http_nifi_1
342aa492510f   apache/nifi:latest   "../scripts/start.sh"    About an hour ago   Up About an hour   8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49179->8080/tcp, :::49179->8080/tcp                     http_nifi_3

There must be three NiFi containers running because we defined the scale count for NiFi to three. Note any of the host ports mapped to 8080. In my case, I have 49177, 49178, and 49179 are mapped to the 8080 port of each NiFi nodes. Wait for some time for all the nodes to detect each other and to form the cluster. Visit http://localhost:49177/nifi in your browser. Instead of 49177, use any host ports assigned by Docker in your machine. You can use any of them because all NiFi nodes can provide you access to the NiFi dashboard.

Run Apache NiFi Cluster in Docker


Step 4:
Though everything works, checking the random port assigned to the NiFi nodes every time you start the cluster is tedious. Let's set a load balancer using Nginx so that we get the same URL all the time.

Step 4.1:
To start with, create a new file named nginx.conf next to the docker-compose.yml with the following content:

user nginx;

events {
    worker_connections 1000;
}
http {
     upstream nifi_server {
        server nifi:8080;
        ip_hash;
    }

    server {
        listen 8080;
        location / {
            proxy_set_header Host "localhost:8080";
            real_ip_header X-Real-IP;
            proxy_pass http://nifi_server;
        }
    }
}

Above Nginx configuration receives the requests at localhost:8080 and redirect them to any NiFi instances running in the same Docker Compose setup. Note that the NiFi UI can be accessed from any nodes. However all consequent requests must be sent to the same node for continuous access. Therefore the Nginx load balancer is configured with ip_hash to send all  requests from the same client to the same NiFi server.

Step 4.2:
Modify the docker-compose.yml created in Step 1 by adding the Nginx service as shown below:

version: "3"
services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: 'zookeeper:latest'
    ports:
      - 2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:latest
    ports:
      - 8080
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_SENSITIVE_PROPS_KEY=xxxxxxxxxxxx
  nginx:
    image: nginx:latest
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - nifi
    ports:
      - "8080:8080"


Step 5:
Stop the currently running cluster by pressing Ctrl + C and run the following Docker Command to start the NiFi cluster with Nginx load balancer.

docker-compose up --scale nifi=3

Wait for some time for all the nodes to detect each other and to form the cluster. Now visit, http://localhost:8080/nifi.

To verify our cluster has all three nodes, click on the hamburger menu and select "Cluster". It will show you all three nodes connected to the cluster.

Run Apache NiFi Cluster in Docker

If everything goes as expected, congratulations! Your NiFi cluster is ready to take more load. However, this cluster is not production ready yet as we don't have secured the cluster via HTTPS and the Zookeeper can be a single point of failure. In the next article, I will show you how to create a NiFi container with SSL enabled.

If you find this article useful or if you have any questions, please leave a comment below.

Previous
Next Post »

Contact Form

Name

Email *

Message *