Rolling Log Files with Logback

 Imagine you are developing a long running service in Java (most likely you already have one) and found a bug in production at some point, how can you identify the root cause of the bug? The answer is simple: analyze your logs. If you already had your logs written to a file you are in a good position. If not, you may have to rely on the mercy of Docker container logs (if your service is containerized) or else you are doomed. 

Even if you have logs appended to a log file, you must consider rolling log files to keep them small and concise. If not, be prepared to open a megabyte/gigabyte size file with millions of lines and find a needle in a haystack. Enabling rolling file log in your application is not a big deal if you know the configuration. However finding the right configuration every time you need it requires a lot of web surfing.

Read More

Run Apache NiFi Docker on Mac M1

In my recent articles on Apache NiFi, I explained how to run Apache NiFi on Docker and Docker Compose. Recently I got a Mac Book M1 Pro machine from work and surprisingly I couldn't run Apache NiFi on Docker in Mac Book M1. Though the Apache NiFi binary deployment works fine on Mac M1 architecture, the official Apache NiFi Docker image does not support Mac M1 yet (at the time of writing this article). However, Chris Sampson a NiFi committer provided a script to build NiFI docker image that is compatible with Mac M1. This article explains, how to build Apache NiFi docker image on your Mac Book M1 and how to run it.

Disclaimer: Since this is not an official image, I recommend this method only for pipeline development and testing purposes.

Read More

Manage GitHub Actions Artifact Storage

If you are reading this post, I hope you have already faced the error: "Create Artifact Container failed: Artifact storage quota has been hit. Unable to upload any new artifacts". If not and if you are about to set up GitHub actions for your repository, read this article before you start to avoid facing the above error down the line.

Image Credits: Pixabay@Pexels

GitHub: the famous code hosting platform stretches beyond being just a hosting platform. GitHub Actions is one such addition that automates workflows in a repository. Though GitHub is generous enough to provide free storage and computing power, there is a limit on your free meal. For more details about the quota, refer to the official document.

Product	Storage	Minutes (per month)
GitHub Free	500 MB	2,000
GitHub Pro	1 GB	3,000
GitHub Free for organizations	500 MB	2,000
GitHub Team	2 GB	3,000
GitHub Enterprise Cloud	50 GB	50,000

GitHub Quota

This article focuses on the storage limitation though similar precaution has to be taken on computation too. Let's say you have a pretty active development team or a community that keeps pushing changes every hour. Suppose your workflow builds an uber jar and packs them into docker containers using two workflow operations; you may choose to upload the uber jar to the artifact storage at the end of the build operation and download it in the deploy operation. However, if your account is a GitHub Free account (with a 500 MB storage limitation) and your uber jar size is around 100 MB, you can only have five artifacts in the artifact storage. If you didn't clear the previous builds, your sixth build will fail to upload the artifact with the following error: Create Artifact Container failed: Artifact storage quota has been hit. Unable to upload any new artifacts.

Read More

Install Oracle JDK 18 on Linux

Even though OpenJDK is available in Linux repositories, some applications strictly require Oracle Java Development Kit. This article shows you how to manually install Oracle JDK $java_version on your Linux system. This article uses JDK $java_version$java_update_no to demonstrate the installation. In the provided commands, replace the version specific paths and file names according to your downloaded version.

Version specific installation guides are available here:

• Install Oracle JDK 8 on Linux
  
• Install Oracle JDK 11 on Linux
• Install Oracle JDK 17 on Linux
• Install Oracle JDK 18 on Linux (The latest version)
  
• Install the latest Oracle JDK on Linux (Permanent link to install the latest version)
  

Oracle provides deb and rpm installers

If your Linux distribution is using DEB package format like Debian, you can download and install the jdk-$java_version$java_update_no_linux-x64_bin.deb file using the following command:

sudo dpkg -i jdk-$java_version_linux-x64_bin.deb

If your  Linux distribution is using RPM package format like Cent OS, you can download and install the jdk-$java_version_linux-x64_bin.rpm file using the following command:

sudo rpm -ivh jdk-$java_version_linux-x64_bin.rpm

However, this article explains the manual installation method which is applicable for all Linux distributions out there. Personally, I prefer the manual installation because I have more control over the changes made in the system.

Read More

Run Apache NiFi Cluster in Docker with SSL Enabled

Welcome to the fourth article in the series of Apache NiFi. The last article explained how to set up an Apache NiFi Docker container with a self-signed SSLcertificate. This article addresses the next pain point: how to create an Apache NiFi cluster in Docker with SSL enabled. Unlike HTTP cluster, setting up Apache NiFi cluster with SSL enabled in Docker introduces a new challenge: Hostname verification.

For added security, if HTTPS connection is enabled, Apache NiFi will verify the Hostname of requests. Therefore each request sent to Apache NiFi must have a predefined hostname. Not only the external requests, but peer-to-peer communication of NiFi nodes in a cluster also go through HTTPS and are subject to hostname verification. If the hostname provided in the HTTPS request does not match the hostname defined in the SSL certificate, NiFi will throw a javax.net.ssl.SSLPeerUnverifiedException.

If you are traditionally deploying Apache NiFi: individual servers with known IP addresses, it is easy to create certificates with those IP addresses. However, in a dynamic environment like Docker, the hostname of a container is defined at the runtime if you need flexible scaling options. Since Docker doesn't provide an option to define the hostname pattern in a scalable cluster, we have to stick to hard-coded Apache NiFi containers with predefined hostnames to create a cluster. The disadvantage of this method is that it is hard to scale up/down a cluster with hard-coded containers. Instead, you can also set up an HTTP cluster and create a load balancer with HTTPS frontend and SSL Termination between the client and NiFi UI. However, in this article, we will stick to the SSL configuration at the cluster level.

Read More

Run Apache NiFi in Docker with SSL Enabled

The last two articles in the Apache NiFi series discussed how to run Apache NiFi standalone server and NiFi cluster in Docker. However, those are far from production-ready because they are not secured. The next step in setting up a secured NiFi cluster is spinning up an Apache NiFi instance with SSL enabled in Docker. Though we are moving towards production-ready, this article will use self-signed certificates. In production, you should not use a self-signed certificate. In addition, you may also require additional safety measures like firewall and proxy.

Read More

Run Apache NiFi Cluster in Docker

The last article on Apache NiFi: Run Apache NiFi in Docker was for those who want to start playing with Apache NiFi. Though it was a good start to play with NiFi, it is far from production deployment. This article introduces the second stage of deployment: a NiFi cluster running in Docker using Docker Compose.

To begin with, you must have Docker installed in your system and also install Docker Compose as we are going to use Docker Compose to setup the Apache NiFi cluster. 

Read More

Resume Tips for Software Engineers in North America

Years ago as an international student preparing my resume to get my first job in Canada, I had a lot of questions and I did search a lot on how to make a resume that fits Canadian employer's requirements and style. Things have changed. Eventually, I got offers from some high tech companies and landed in a good job and now I am interviewing candidates who are like I was a couple of years ago. After looking at more and more resumes, I decided to share my experience here with a hope that will make someone's life better.

For whom this article is for? Well! for anyone looking for a new job in Canada. Especially if you are  new immigrant who has no idea about what Canadian employers are looking for, this article is tailored to your requirements. Though my experience is limited to Canada the companies I applied for are mostly US-based companies so I hope it can be applied anywhere in North America. This article is targeting only those who are in the software industry. I don't know how much it will overlap with other industries. The rest of the article is divided into two topics: 1. Resume Sections, 2. Resume Format. The first topic covers what to include and not to include in your resume and the second topic provides some formatting tips to make your resume get you a call from the recruiter.

Read More