Integrating Angular with OpenCV.js with ng-open-cv

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library. It was originally written in C++ and has since been ported to most other major languages and platforms and most recently, with the boost provided by Web Assembly, it has finally crossed into the Javascript world. OpenCV.js is a “JavaScript binding for selected subset of OpenCV functions for the web platform”, and as you have noted, it exposes a subset of the OpenCV functionality to the web. I ran into this library trying to answer the question:

How do I detect a face in a picture using Javascript?

Initially, I went for Tracking.js a promising library that implemented a face detection algorithm that works pretty well, but the event based natured of the implementation, meaning that found faces are emitted as events, rather than the library being able to deterministically tell you whether it found a face or not made it challenging to work with in my particular use case where I wanted to know if a face was found as fast as possible. Furthermore, on top of face detection, I wanted to be able to detect an ID card (Driver License)in an image, which Tracking.js does not support. Googling around, OpenCV just kept coming back as the solution for similar use cases, so I finally decided to use it for all my image feature detection features. In these days there is an integration for all major libraries, or so I thought, imagine my surprise googling for an Angular OpenCV integration library and coming up with… fohi (nothing in Dyula )!!! The next paragraph will explain why.

Working with OpenCV.js in Angular

The tutorial explains that

OpenCV.js leverages Emscripten to compile OpenCV functions into asm.js or WebAssembly targets, and provides a JavaScript APIs for web application to access them

Existing library absent, I figured, let me just download the distribution, integrate it in my @angular/cli configuration and I am home free. WRONG! There is no official distribution or NPM package available for the Web, unless you’re working with NodeJS in the backend . The official usage guide and and al tutorials keeps referring to an opencv.js library file but in the example pages it is part of the build of the page. This meant I was going to have to build it from source as explained in the official documentation . Here are the steps to follow to building your own local version if you wanted to:

Install EmScripten

Emscripten is an LLVM-to-JavaScript compiler. In English, Emscripten basically allows you to compile C/C++ code into Javascript. This allows native code to be run on the web at near native performance, if you’ve ever heard of Web Assembly, this is what the hoopla is about . In our specific case, something as computationally intensive as computer vision can be brought to the Web using Javascript and Emscripten helps us get there. To install:

# Get the emsdk repo
git clone https://github.com/juj/emsdk.git

# Enter that directory
cd emsdk

# Download and install the latest SDK tools.
./emsdk install latest

# Make the "latest" SDK "active" for the current user. (writes ~/.emscripten file)
./emsdk activate latest

# Activate PATH and other environment variables in the current terminal
source ./emsdk_env.sh

On Windows, run <code>emsdk</code> instead of <code>./emsdk</code>, and <code>emsdk_env.bat</code> instead of <code>source ./emsdk_env.sh</code>.

Build OpenCV

Next, we need to build OpenCV. You will need Python (≥ 2.7):

# Install from Github.
git clone https://github.com/opencv/opencv.git

<strong># Go the openCV folder and run the build.</strong>

cd opencv

python ./platforms/js/build_js.py dist

The official guide says that:

The build script builds asm.js version by default. To build WebAssembly version, append --build_wasm switch.

I’ve found that to be incorrect. It always builds the WebAssembly version. I did not mind since the WASM version brings near native performance. Once the build is done, it takes a while, at the root of your folder you should have a dist/bin folder with 3 files

opencv_js.js
opencv_js.wasm
opencv.js

It wasn’t clear to me and I’ve researched it what the is for but the two that we’re interested with are opencv.js and opencv_js.wasm files. The files are created to be distributed so in the case of the opencv.js file, the file tripped my VSCode editor as some of the JS code was in a single line that exceeds the max line length, making Intellisense trip and crashing the plugin. Due to this line break issue, this will also break your @angular/cli build if you integrate it as is. I had to import the file in a Javascript editor to be able to format it correctly and be able to view it in VSCode.

Working with OpenCV.js

These initial issues outstanding, I ran into a bunch of issues trying to integrate into my Angular project. OpenCV.js as great as it is, is only documented through tutorials that assume that your initial configuration has already been done. It made it very difficult to figure out exactly what to do and it took me long tentative approach to finally get it working in a stable way.

Adding opencv.js to angular-cli.json

I added a new entry to the scripts:[] property in my angular-cli.json. On ng serve I ran into a

failed to load wasm binary file at opencv_js.wasm

error. I had to dig through the source code to be able to understand the source of the error, which of course was specifying the correct path to the opencv_js.wasm file, which for reference, I put in my assets folder. The library can’t function without that file being loaded. This brings me down to my next point.

The Module global variable

To make a long story short, as of version 3.4.3 (and v4 I am pretty sure):

The WASM OpenCV.js relies on a global Module variable, that has to be present and configured before the OpenCV.js library is loaded.

I’ve found that out doing a lot of view source on code examples. A minimal configuration should look like this:

var Module =  {

wasmBinaryFile: 'wasm/opencv_js.wasm',

usingWasm: true,

onRuntimeInitialized: () => { console.log('OpenCV is ready)}

}

As long as such an object is present on the page before the library is loaded, OpenCV.js will boot fine. It feels icky and very much in my opinion against the best practices for today’s libraries and I hope this is something that will be addressed in future releases of the library. Until then, pay attention to the wasmBinaryFile property in particular, it should point to the location in your project where the WASM file should be accessible. In my case it would have been assets/opencv/wasm/opencv_js.wasm . There are two ways you can go about including this in your project:

• Adding a var Module in a <script> tag in your index.html with the right configuration.
• Add an opencv_init.js file to your codebase, either in your src or assets folders, and add the correct path to your angular-cli.json scripts property. You have to make sure that file entry comes before the opencv.js file if you are loading the opencv.js library file through Webpack like:

"scripts": [
...

"../src/opencv/opencv_init.js",
"../src/opencv/opencv.js"

],

In any case, with this setup in place, you should be good to go and be able to use the library now.

Module Configuration

As explained before, there are two configuration properties in the opencv Module you will need to pay attention to:

• wasmBinaryFile  : This is the location of the WASM file associated with OpenCV. I am guessing if you are successful in building the ASM version you would not need to worry about that. The associated usingWasm property did not make a difference whether I set it to true or false. It would always try to load the WASM.
• onRuntimeInitialized  : This function is the callback triggered when the library is ready for use. If you’re integrating OpenCV through your own Angular service, you would need to wait until this callback is called to make the OpenCV based functionality available to the rest of your code. I have taken this into account by wrapping every of my OpenCV calls into a subscription to a isReady$ BehaviorSubject that will emit only when the library is ready.

Considerations for Face Detection

As stated in the beginning of this piece, my original goal for working with OpenCV was to make use of their face detection functionality. If you look at the example code here , and tried to copy and use it in your code, it would fail on this line:

faceClassifier = new cv.CascadeClassifier();
faceClassifier.load('haarcascade_frontalface_default.xml');
  
eyeClassifier = new cv.CascadeClassifier();
eyeClassifier.load('haarcascade_eye.xml');

Those files are part of the data folder that comes with the release and are the classification data needed by the ML algorithm uses to return result. Trying to run the code as is will get you an error like:

ERROR 6424592 - Exception catching is disabled, this exception cannot be caught. Compile with -s DISABLE_EXCEPTION_CATCHING=0 or DISABLE_EXCEPTION_CATCHING=2 to catch.

when using the detectMultiScale function. I’ll make a long story short here by saying that these files need to be created in memory and OpenCV.js has an non-advertised utility class called Utils that can be used for exactly that and other useful operations. I’ve used that class as the basis for my integration library, which saves you the whole pains I’ve just went through.

Announcing ng-open-cv

This is an open source integration of OpenCV.js and Angular 6 and it works currently as a service exposing various OpenCV functionality. It’s based on an “angularized” version of the Utils class provided by the framework and will allow you initially to:

• Load OpenCV.js v3.4.3 asynchronously
• Load classifier files

This is for the alpha release. Eventually, I am hoping to add functions to facilitate:

• Face detection
• Corner detection
• Blur detection

And any other useful functions that the community will contribute to of course. You can grab the library here and I hope it will be useful to you as it was to me.

That was quite some reading, reward yourself with some Yang System!