OpenCV and Node.js - Image processing

This is the third part of image processing series . We will create REST API’s for smoothing, threshold filters, finding contour in an Image and integrating with thePart-2 of the series.

Note: If you have not readpart 2 then pls head over to it. Read and implement it and then come back to this article. For those who still skipped part 2, here is a lil gist.

In Part 2 we coded the UI in React.js for each filter and implemented a live image captured using the webcam. The only part left was to code the callable API in the UI.

The final UI looks like this

Note: Before we start here is the link of Github repo of the code we will implement - https://github.com/overflowjs-com/image_app_opencv_api_part3

Let’s start,

We will create an endPoint /api/apply_filter , whose schema looks like

Method:  POST
Data-Type: application/json
Request payload: {
  data: "base64 encoded image data",
  type: "Type of filter to be applied"
}

To jump-start the project we will use https://github.com/developit/express-es6-rest-api which is a sample API starter kit with everything setup. Feel free to use any of the boilerplate with ES6 tooling.

1. Clone the repo using below

git clone https://github.com/developit/express-es6-rest-api.git image_app_api_part_3

2. Install initial packages

cd image_app_api_part_3npm i

3. Open config.json in project root change port from 8080 to 4000, and body limit to 10000kb.

4. Open src > api > index.js file and add the replace it with below code

import { version } from '../../package.json';
import { Router } from 'express';
import facets from './facets';
 
export default ({ config, db }) => {
    let api = Router();
 
    // mount the facets resource
    api.use('/facets', facets({ config, db }));
 
    // perhaps expose some API metadata at the root
    api.get('/', (req, res) => {
        res.json({ version });
    });
 
    api.post('/apply_filter', (req, res) =>  {
        return res.json({ "msg": "hello"})
    });
 
    return api;
}

We only added a POST endpoint /apply_filter which for now returns { msg: "hello" }

5. Now, let’s install OpenCV module `opencv4nodejs`, Please refer https://www.npmjs.com/package/opencv4nodejs for the installation guide on mac/Linux/windows and make sure it’s properly installed.

6. We will now add some smoothing filters, go to src > api and create a folder named filters , inside it create a new file named FilterProcessor.js

import cv from 'opencv4nodejs';   
import SmoothingFitlers from './SmoothingFilters';
import ThresholdingFilters from './ThresholdingFilters';
import ContourFilters from './ContourFilters';
export default class FilterProcessor {
    constructor(data, type) {
        this.data = data;
        this.type  = type
    }
getInputImage() {
const base64data =this.data.replace('data:image/jpeg;base64','')
                            .replace('data:image/png;base64','');//Strip image type prefix
        const buffer = Buffer.from(base64data,'base64');
        const image = cv.imdecode(buffer);
return image;
        
    }
process() {
        
        let outputImage = null;
if (["blur", "gaussian_blur", "median_blur", "bilateral_filter"].indexOf(this.type) >  -1) {
            const filter = new SmoothingFitlers(this.type, this.getInputImage());
            outputImage = filter.process();
        }
        
        const outBase64 =  cv.imencode('.jpg', outputImage).toString('base64');
const output = 'data:image/jpeg;base64,'+outBase64 + ''
return output;
    }
}

In this class, there are two values passed in constructor — Data and Type. Data is a base64 encoded image. Type is the filter or action that we need to do on the image.

We have two functions-

getInputImage()
process()

We are checking the type of filter that lies in [“blur”, “gaussian_blur”, “median_blur”, “bilateral_filter”]. We are initiating SmoothingFilter class with two values in its constructor filter type and cv: Mat image of the available image and calling a function process() on the filter and saving output Image.

7. Below code in SmoothingFilter.js is converting Mat object to base 64 and return the base 64 filtered image.

import cv from 'opencv4nodejs';
export default class SmoothingFilters {
constructor(type, image) {
        this.type = type;
        this.image = image;
    }
process() {
        let processedImage = null;
if (this.type == "blur") {
            processedImage = cv.blur(this.image, new cv.Size(10, 10));
        } else if(this.type == "gaussian_blur") {
            processedImage = cv.gaussianBlur(this.image, new cv.Size(5, 5), 1.2, 1.2);
            // processedImage = this.image.gaussianBlur(new cv.Size(10, 10), 1.2);
        } else if(this.type == "median_blur") {
            processedImage = cv.medianBlur(this.image, 10);
        } else if(this.type == "bilateral_filter") {
            processedImage = this.image.bilateralFilter(9, 2.0, 2.0);
        }
return processedImage;
    }
}

This is a smoothing filter class takes two-parameter in constructor — the type of filter and source image Mat object.

Inside process function based on each filter type we are processing the image and returning the processed image.

Note: For more details on each parameter in each of these filters go to — https://bit.ly/2H6f5N4

This documentation link and try multiple parameters it will help you understand what each of these smoothing filters do.

8. Let’s now add function to the router and test it. So, if you go in to src > index.js

import { version } from '../../package.json';
import { Router } from 'express';
import facets from './facets';
import FilterProcessor from './filters/FilterProcessor';
 
export default ({ config, db }) => {
    let api = Router();
 
    // mount the facets resource
    api.use('/facets', facets({ config, db }));
 
    // perhaps expose some API metadata at the root
    api.get('/', (req, res) => {
        res.json({ version });
    });
 
    api.post('/apply_filter', (req, res) =>  {
 
        // console.log(req.body);
 
        const data = req.body.data;
        const type = req.body.type;
 
        const processor = new FilterProcessor(data, type);
 
        return res.json({type: type, data: processor.process()});
    });
 
    return api;
}

Here, we have fetched the data and type from the request body, which is passed by our UI code and then call FilterProcessor function with data and type and returned the processed data and type back as a response.

Here is how the UI looks like

As effect are visible but with some algos, there is not too much change we see. But yes each has responded properly.

9. Now let’s add another set of filters for thresholding. We will create a new class ThresholdingFilter.js inside src > filters .

import cv from 'opencv4nodejs';
export default class ThresholdingFilters {
constructor(type, image) {
        this.type = type;
        this.image = image;
 }
process() {
        let processedImage = null;
this.image = this.image.gaussianBlur(new cv.Size(5, 5), 1.2);
        this.image = this.image.cvtColor(cv.COLOR_BGR2GRAY);
            
        if (this.type == "simple_threshold") {
            processedImage = this.image.threshold(127, 255, cv.THRESH_BINARY);
        }
            
        if(this.type == "adaptive_threshold") {
            processedImage = this.image.adaptiveThreshold(255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 11, 2);
        }
if(this.type == "otasu_threshold") {
            processedImage = this.image.threshold(0, 255, cv.THRESH_BINARY+cv.THRESH_OTSU);
        }
return processedImage;
        
    }
}

The constructor is accepting common values cv: Mat object (image) and type of filter. In the process() , we have implemented various thresholding filters.

Something to see

this.image = this.image.gaussianBlur(new cv.Size(5, 5), 1.2);
this.image = this.image.cvtColor(cv.COLOR_BGR2GRAY);

In the above two lines, we have blurred the image using gaussian blur to reduce noise on the input image and in the next line, we have converted an image to grayscale.

Adaptive thresholding works with 8UC1 format which means it works for only one channel and the grayscale image has only one channel that is black.

Next based on the type we are operating thresholding techniques.

Note: For more details on parameters and thresholding techniques go to- https://docs.opencv.org/3.4.0/d7/d4d/tutorial_py_thresholding.html

In FitlerProcessor process, we have added an array of filter and called ThresholdingFilters.js function.

if (["simple_threshold", "adaptive_threshold", "otasu_threshold"].indexOf(this.type) > -1) {
            const filter = new ThresholdingFilters(this.type, this.getInputImage());
            outputImage = filter.process();
}

11. Now the last implementation left is finding contour in the image. We will now add a new class ContourFilter.js inside filter folder.

import cv from "opencv4nodejs";
export default class ContourFilters {
constructor(type, image) {
        this.type = type;
        this.image = image;
    }
process() {
this.image = this.image.gaussianBlur(new cv.Size(5, 5), 1.2);
let grayImage = this.image.cvtColor(cv.COLOR_BGR2GRAY);
        grayImage = grayImage.adaptiveThreshold(255, cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 11, 2);
if(this.type == "find_all_contours") {
            
            let contours = grayImage.findContours(cv.RETR_TREE, cv.CHAIN_APPROX_NONE, new cv.Point2(0, 0));
const color = new cv.Vec3(41, 176, 218);
contours = contours.sort((c0, c1) => c1.area - c0.area);
const imgContours = contours.map((contour) => {
                return contour.getPoints();
            });
this.image.drawContours(imgContours, -1, color, 2);
        }
if(this.type == "find_filtered_contours") {
let contours = grayImage.findContours(cv.RETR_LIST, cv.CHAIN_APPROX_NONE,  new cv.Point2(0, 0));
            
            const color = new cv.Vec3(41, 176, 218);
            
            contours = contours.sort((c0, c1) => c1.area - c0.area);
const imgContours = contours.map((contour) => {
                return contour.getPoints();
            });
this.image.drawContours(imgContours, -1, color, 0);
        }
return  this.image;
    }
}

The implementation style remains the same as other filters i.e a constructor function and a process function. What varies is the process function implementation. In the process function —

this.image = this.image.gaussianBlur(new cv.Size(5, 5), 1.2);
let grayImage = this.image.cvtColor(cv.COLOR_BGR2GRAY);
grayImage = grayImage.adaptiveThreshold(255,cv.ADAPTIVE_THRESH_GAUSSIAN_C, cv.THRESH_BINARY, 11, 2);

We are blurring the image, converting it to grayscale and then thresholding it, to use it for contour finding.

There are two methods we have used in each filter findContour() and drawContour , which help in finding edges or boundaries and then drawing it to the main colored image. Also, the operations are performed on a threshold image and drawn on a colored image.

Note: Read about image contour below- https://docs.opencv.org/2.4/modules/imgproc/doc/structural_analysis_and_shape_descriptors.html?highlight=findcontours#findcontours

Inside FiltereProcessing process function,

if (["find_all_contours", "find_filtered_contours"].indexOf(this.type) >  -1) {
      const filter = new ContourFilters(this.type, this.getInputImage());
      outputImage = filter.process();
}

and we are done :)

Feel free to play around with the code — https://github.com/overflowjs-com/image_app_opencv_api_part3

Final words:

This series is enough to start working on any scale of image processing and computation project. We have used OpenCV which is one of the most powerful images processing framework.

There are tons of example we have for Nodejs- https://github.com/justadudewhohacks/opencv4nodejs/tree/master/examples . Try to build them and have fun with them.

Now there are two important keys in image processing

1. Image computation

2. Image storage

In this basic example, we are using a very low-resolution image from my webcam 640x480. This project may not work properly if you will increase the resolution to HD or higher. For it, we need to have some other implementation setup in backend.

If you have looked Cloudinary you can get and use the idea from them. On each API call, we are passing base64 data. And also we are returning base64 processed data. For small payload and image, these API will work great but resolution of 2mb, 4mb or more passing base64 data is not feasible.

So what we can do is to store images in the backend in any storage system. Generate filtering in it save filtered image on the same storage system and return filtered image URL. It will release the pain of passing base64 data.

Thanks for Reading hope you will be enabled by this series to do anything in React.js with image processing in hand.

Note: There are various bugs with opencv4nodejs keep referring original OpenCV documentation as their documentation is also buggy in many cases.

Get yourself added to our 2500+ people subscriber family to learn and grow more and please hit the share button on this article to share with your co-workers, friends, and others.

Check out articles onJavascript, Angular , Node.js , Vue.js .

For more articles stay tuned tooverflowjs.com

Thank you!