The last article on Apache NiFi: Run Apache NiFi in Docker was for those who want to start playing with Apache NiFi. Though it was a good start to play with NiFi, it is far from production deployment. This article introduces the second stage of deployment: a NiFi cluster running in Docker using Docker Compose.
To begin with, you must have Docker installed in your system and also install Docker Compose as we are going to use Docker Compose to setup the Apache NiFi cluster.
Step 1:
Create a new file named docker-compose.yml with the following content:
version: "3"
services:
zookeeper:
hostname: zookeeper
container_name: zookeeper
image: 'zookeeper:latest'
ports:
- 2181
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
nifi:
image: apache/nifi:latest
ports:
- 8080
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
- NIFI_SENSITIVE_PROPS_KEY=xxxxxxxxxxxxIn the above configuration, change the value of the NIFI_SENSITIVE_PROPS_KEY with a strong 12 character password to encrypt the sensitive properties used in the NiFi configuration. Though the provided key works, it is not secured enough.
Step 2:
Run the following Docker command to run 3 instances of Apache NiFi. Note that
NiFi instances can communicate between themselves with the help of Zookeeper
and elect a node as the Cluster Coordinator. For more details about the
Zero-Leader Clustering employed in Apache NiFi, check the
official document.
docker-compose up --scale nifi=3
Step 3:
Run the following Docker command to list the currently running Docker
containers.
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
df73fe7dfad2 zookeeper:latest "/docker-entrypoint.…" About an hour ago Up About an hour 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:49180->2181/tcp, :::49180->2181/tcp zookeeper
e1ebfa5fe7e0 apache/nifi:latest "../scripts/start.sh" About an hour ago Up About an hour 8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49177->8080/tcp, :::49177->8080/tcp http_nifi_2
455cb9f8ff1f apache/nifi:latest "../scripts/start.sh" About an hour ago Up About an hour 8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49178->8080/tcp, :::49178->8080/tcp http_nifi_1
342aa492510f apache/nifi:latest "../scripts/start.sh" About an hour ago Up About an hour 8000/tcp, 8443/tcp, 10000/tcp, 0.0.0.0:49179->8080/tcp, :::49179->8080/tcp http_nifi_3There must be three NiFi containers running because we defined the scale count for NiFi to three. Note any of the host ports mapped to 8080. In my case, I have 49177, 49178, and 49179 are mapped to the 8080 port of each NiFi nodes. Wait for some time for all the nodes to detect each other and to form the cluster. Visit http://localhost:49177/nifi in your browser. Instead of 49177, use any host ports assigned by Docker in your machine. You can use any of them because all NiFi nodes can provide you access to the NiFi dashboard.
Step 4:
Though everything works, checking the random port assigned to the NiFi nodes
every time you start the cluster is tedious. Let's set a load balancer using
Nginx so that we get the same URL all the time.
Step 4.1:
To start with, create a new file named nginx.conf next to the
docker-compose.yml with the following content:
user nginx;
events {
worker_connections 1000;
}
http {
upstream nifi_server {
server nifi:8080;
ip_hash;
}
server {
listen 8080;
location / {
proxy_set_header Host "localhost:8080";
real_ip_header X-Real-IP;
proxy_pass http://nifi_server;
}
}
}
Above Nginx configuration receives the requests at localhost:8080 and redirect
them to any NiFi instances running in the same Docker Compose setup. Note that the NiFi UI can be accessed from any nodes. However all consequent requests must be sent to the same node for continuous access. Therefore the Nginx load balancer is configured with ip_hash to send all requests from the same client to the same NiFi server.
Step 4.2:
Modify the docker-compose.yml created in Step 1 by adding the Nginx service as
shown below:
version: "3"
services:
zookeeper:
hostname: zookeeper
container_name: zookeeper
image: 'zookeeper:latest'
ports:
- 2181
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
nifi:
image: apache/nifi:latest
ports:
- 8080
environment:
- NIFI_WEB_HTTP_PORT=8080
- NIFI_CLUSTER_IS_NODE=true
- NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
- NIFI_ZK_CONNECT_STRING=zookeeper:2181
- NIFI_ELECTION_MAX_WAIT=1 min
- NIFI_SENSITIVE_PROPS_KEY=xxxxxxxxxxxx
nginx:
image: nginx:latest
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- nifi
ports:
- "8080:8080"Step 5:
Stop the currently running cluster by pressing Ctrl + C
and run the following Docker Command to start the NiFi cluster with Nginx load
balancer.
docker-compose up --scale nifi=3Wait for some time for all the nodes to detect each other and to form the cluster. Now visit, http://localhost:8080/nifi.
To verify our cluster has all three nodes, click on the hamburger menu and select "Cluster". It will show you all three nodes connected to the cluster.
If everything goes as expected, congratulations! Your NiFi cluster is ready to take more load. However, this cluster is not production ready yet as we don't have secured the cluster via HTTPS and the Zookeeper can be a single point of failure. In the next article, I will show you how to create a NiFi container with SSL enabled.
If you find this article useful or if you have any questions, please leave a
comment below.
EmoticonEmoticon