Image Processing in React.js (Part 1) – Creating Cool Filter

Image processing and manipulation has been interesting field to work from the beginning. I had done a wide variety of work in image processing application from creating basic image filters to Augmented Reality apps.

We will implement basic features for image processing today.

Brief about Pixel and Image

Every digital imprint of the image contains pixels. Each pixel determined by some values. There are different mechanisms to define and store these pixel values. Pixel value can be represented with RGB, RGBA.

Each pixel comprises of color code values and in combination, it represents a dot. When you connect multiple dots entire image is built. While many image processing systems manipulate over these values and provides some cool filters which you might have seen on Instagram too.

Each value in a pixel can be said as a channel. Transparency channel also termed as A (Alpha) contains transparency details. The difference between RGB and RGBA is due to the transparency channel.

Colorful images use three or four channels depending on the image format.

Note: There are a lot of libraries for image processing, out of which OpenCV , imageMagick provide a wide variety of algorithms to do image manipulation and processing. Starting from creating masks over the image to do image processing like Object detection and extraction. Motion detection Image processing on videos done in a similar way.

As the evolution part javascript provide wide majority of libraries to implement and play with images.

We will use Cloudinary ( https://cloudinary.com ) for image manipulation and react for UI for this project.

Live App link of what we will create today - https://cryptic-sierra-27182.herokuapp.com/ , if you want to play around a bit.

Prerequisite

1. Node version >= 9.2.1

2. Create react app - https://facebook.github.io/create-react-app/docs/getting-started

3. Some knowledge around material UI as we will use it for UI design in our react app- https://material-ui.com

Let’s create a react app and install the dependencies

1. Create a react app named image_app

   create-react-app image_app

1. Go to created directory and install cloudinary-react sdk for processing images

   cd image_app
   npm install cloudinary-react --save

2. Next is to install Material UI theme for getting some quick controls to manipulate over images

   npm install @material-ui/core

Directory Structure

Now we have our dependencies installed and basic project directory setup, which looks like below

image-app
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   └── manifest.json
└── src
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    └── serviceWorker.js

We will now, create a container folder inside the src folder, and add ImageOps.jsx file to it which will contain our UI code to build our image processing filters.

image-app
├── README.md
├── node_modules
├── package.json
├── .gitignore
├── public
│   ├── favicon.ico
│   ├── index.html
│   └── manifest.json
└── src
    ├── Container
    │   ├── ImageOps.jsx <------------this
    ├── App.css
    ├── App.js
    ├── App.test.js
    ├── index.css
    ├── index.js
    ├── logo.svg
    └── serviceWorker.js

And in App.js we will call the ImageOps file as below.

Note: You can see the entire project on overflowjs Github repo - https://github.com/overflowjs-com/image_app_cloudinary_material_react

Now basic setup is done, we will create some RGB settings, HSV settings and Advance filter on image inside ImageOps.jsx file.

RGB Settings

An image can be represented with RGB values per pixels. Each pixel contains a set of values that will determine how the image will be displayed. We will provide a mechanism to change each of these channel values.

We have used material Grid to distribute our overall UI parts:

1. We will build first row with two columns, One to display original image and another to show altered image.

<Grid item xs={6}>
    <Card>
        <CardContent>
            <Typography variant="body2" color="textSecondary" component="p">
                Input image
            </Typography>
            <Image publicId="leena" cloudName="rakesh111" >
            </Image>
        </CardContent>
    </Card>
</Grid>
<Grid item xs={6}>
    <Card>
        <CardContent>
            <Typography variant="body2" color="textSecondary" component="p">
                Output Image
            </Typography>
            <Image publicId="leena" cloudName="rakesh111" >
                {this.getTransformations()}
            </Image>
        </CardContent>
    </Card>
</Grid>

Each grid contain, card and heading mentioned as Typography which is imported from material UI. The Image component above is imported from cloudinary dictionary. These attributes are fetched from libraries as shown below.

import Grid from '@material-ui/core/Grid';
import Card from '@material-ui/core/Card';
import Typography from '@material-ui/core/Typography';
import {Image} from 'cloudinary-react';

The output image code call’s getTransformations function and ask for current transformation to be applied as per selected transformation on UI. We keep our selected transformation in react state.

getTransformations() {

    return this.state.transforms.map((tranform) => {

        return ( <Transformation effect={`${tranform.key}:${tranform.value}`} gravity="center" crop="fill" />)
    })
}

On state change we form a transformations array that contain each change made to an image. Based on each of those transformations which is a key => value pair object, we do a re-render of UI.

These key=>value pairs will create transformation pipeline to render inside Image component of cloudinary.

Let’s create our first initial slider controls for changing RGB values.

Again we use 2 Grid styles UI as we did previously.

<Grid item xs={6}>
    <Card>
        <CardContent>
            <Box color="text.primary">
                <Typography paragraph={true} variant="h5" align="left" component="h5">
                    R-G-B Controls
                </Typography>

                {this.getRBBCons().map((color) => {
                    return (
                        <SliderComponent getSliderValue={(key) => this.getSliderValue(key, "rgb")} default={0} min={-100} max={100} keyLabel={color.key} keyValue={color.value} 
                            updateColorValue={(e, value, key) => this.updateColorValue(e, value, key)}  />
                    )
                })}

                <Button variant="contained" align="left" onClick={() => this.resetFilters(["red", "green", "blue"])} color="primary">
                    Reset
                </Button>
            
            </Box>
        </CardContent>
    </Card>
</Grid>
<Grid item xs={6}>
    <Card item xs={6}>
        <CardContent>
            <Box color="text.primary">
                <Typography paragraph={true} variant="h5" align="left" component="h5">
                    R-G-B Based Filters
                </Typography>
                
                <Button variant="contained" align="left" onClick={() => this.createRGBEffect("all_blue")} >
                    Fill Blue
                </Button>
                <Button variant="contained" align="left" onClick={() => this.createRGBEffect("all_red")} >
                    Fill Red
                </Button>
                <Button variant="contained" align="left" onClick={() => this.createRGBEffect("all_green")}>
                    Fill Green
                </Button>
                <Button variant="contained" align="left" onClick={() => this.resetFilters(["red", "green", "blue"])} color="primary">
                    Reset
                </Button>
            
            </Box>
        </CardContent>
    </Card>
</Grid>

Function getRBBCons will give constant to help transformations over image as below

getRBBCons() {
    return [
        {key: "Red", value: "red", default: 0},
        {key: "Green", value: "green", default: 0},
        {key: "Blue", value: "blue", default: 0}
    ]
}

It provide sliders constants to iterate and create separate slider

getSliderValue(key, type) {
        
    const transform = this.state.transforms.find((transform) => transform.key === key);
    
    if (transform) {
        return transform.value;
    }

    if (type == "rgb") {
        return this.getRBBCons().find((transform) => transform.value === key).default;
    } else if (type == "hsv") {
        return this.getHSVCons().find((transform) => transform.value === key).default;
    }

}

This function provides binding to current effect value for RGB and HSV. Every-time the value of slider is changed it gets updated on current state channel value.

updateColorValue(e, value, key) {
    const transform = {
        key,
        value
    }

    const transforms = this.getUpdatedTransform(this.state.transforms, transform);
    this.setState({transforms});

}

getUpdatedTransform(transforms, transform) {

    const newTransforms = transforms.filter(({key}) => key !== transform.key)
    
    newTransforms.push(transform);

    return newTransforms

}

SliderComponent is basic extension of Slider material UI

class SliderComponent extends React.Component {

    valuetext(value) {
        return `${value}°C`;
    }

    render() {
        return (
            <div>
                <Typography id="discrete-slider" align="left" gutterBottom>
                    {this.props.keyLabel}
                </Typography>
                <Slider
                    defaultValue={this.props.default}
                    getAriaValueText={this.valuetext}
                    aria-labelledby="discrete-slider"
                    valueLabelDisplay="auto"
                    step={10}
                    value={this.props.getSliderValue(this.props.keyValue)}
                    marks
                    min={this.props.min}
                    max={this.props.max}
                    onChangeCommitted={(e, value) => this.props.updateColorValue(e, value, this.props.keyValue)}
                />
            </div>
        )
    }
    
}

With current implementation we can create several strait filters like

1. All Blue with Red and Green set to minimum and Blue to maximum

2. All Red with Blue and Green set to minimum and Red to maximum

3. All Green with Red and Blue set to minimum and Green to maximum

In another Grid we will create buttons for handling these filters. Function of these buttons are below:-

createRGBEffect(type) {
    
    const red = {key: "red", value: 0}
    const blue = {key: "blue", value: 0}
    const green = {key: "green", value: 0}

    switch(type) {
        case "all_red":
            red.value = 100;
            break;
        case "all_blue":
            blue.value = 100;
            break;
        case "all_green":
            green.value = 100;
            break;
        default:
            break;
    }

    let transforms = this.state.transforms;

    transforms = this.getUpdatedTransform(transforms, red);
    transforms = this.getUpdatedTransform(transforms, blue);
    transforms = this.getUpdatedTransform(transforms, green);

    this.setState({transforms})
}

If you want to see the final code of imageOps.jsx - https://github.com/overflowjs-com/image_app_cloudinary_material_react/blob/master/src/Container/ImageOps.jsx

Another representation/manipulation of image is HSV. Let’s look at it.

HSV Settings

We implemented another set of settings to incorporate HSV controls.

Note: For more info around these feature you can refer to - https://cloudinary.com/documentation/image_transformation_reference .

HSV model provides a cylindrical/cone representation of an image. Hue is a 360-degree implementation of Red (0–60), Yellow (61–120), Magenta ( 301–360), Cyan(181–240), Blue (241–300), Green (121–180), where each portion can contribute from given degrees.

Saturation : - Describes amount of grey color, if we reduc it gives we get more grey color.

Value (Brightness) : With saturation it represents strength of color.

Let’s look at the code,

<Grid item xs={6}>
    <Card>
        <CardContent>
            <Box color="text.primary">
                <Typography paragraph={true} variant="h5" align="left" component="h5">
                    H-S-V Controls
                </Typography>

                {this.getHSVCons().map((color) => {
                    return (
                        <SliderComponent getSliderValue={(key) => this.getSliderValue(key, "hsv")} default={0} min={-100} max={100} keyLabel={color.key} keyValue={color.value} 
                            updateColorValue={(e, value, key) => this.updateColorValue(e, value, key)}  />
                    )
                })}

                <Button variant="contained" align="left" onClick={() => this.resetFilters(["hue", "saturation"])} color="primary">
                    Reset
                </Button>
                
            </Box>
        </CardContent>
    </Card>
</Grid>
<Grid item xs={6}>
    <Card item xs={6}>
        <CardContent>
            <Box color="text.primary">
                <Typography paragraph={true} variant="h5" align="left" component="h5">
                    H-S-V Based Filters
                </Typography>
                
                <Button variant="contained" align="left" onClick={() => this.createHSVEffect("grayscale")} >
                    Gray Scale
                </Button>
                <Button variant="contained" align="left" onClick={() => this.createHSVEffect("sepia")} >
                    Sepia
                </Button>
                <Button variant="contained" align="left" onClick={() => this.resetFilters(["hue", "saturation", "brightness"])} color="primary">
                    Reset
                </Button>
            
            </Box>
        </CardContent>
    </Card>
</Grid>

Again HSV filter implementation is done in two parts.

While most of the components are similarly implemented on first gird. We have three slider for Hue, Saturation and Value.

getHSVCons() {
    return [
        {key: "Hue", value: "hue", default: 80},
        {key: "Saturation", value: "saturation", default: 80},
        {key: "Value", value: "brightness", default: 80},
    ]
}

This function gives SliderComponent a way to define constants for HSV.

On second grid we have two buttons Grayscale and Sepia to show the image effects. The main code which handles these button effects are:-

createHSVEffect(type) {
        
    const hue = {key: "hue", value: 80}
    const saturation = {key: "saturation", value: 80}

    switch(type) {
        case "grayscale":
            saturation.value = -70;
            break;
        case "sepia":
            hue.value = 20;
            saturation.value = -20
            break;
        default:
            break;
    }

    let transforms = this.state.transforms;

    // transforms = this.getUpdatedTransform(transforms, hue);
    if(type == "grayscale") {
        transforms = this.getUpdatedTransform(transforms, saturation);
    } else if(type == "sepia") {
        transforms = this.getUpdatedTransform(transforms, hue);
        transforms = this.getUpdatedTransform(transforms, saturation);
    }
    this.setState({transforms})

}

Here we have made some constant values to push the transformations for greyscale and sepia.

Last portion is creating advanced filters provided by cloudinary which is the coolest part here. Let’s look into it.

Advance Filters

Below is the code to make the above UI

<Grid xs={12}>
    <Card item xs={6}>
        <CardContent>
            <Box color="text.primary">
                <Typography paragraph={true} variant="h5" align="left" component="h5">
                    Advance Filters By Cloudinary
                </Typography>
                
                <Button variant="contained" align="left" onClick={() => this.createAdvanceEffects("cartoon")} >
                    Cartoonify
                </Button>
                <Button variant="contained" align="left" onClick={() => this.createAdvanceEffects("vignette")} >
                    Vignette
                </Button>

                <Button variant="contained" align="left" onClick={() => this.createAdvanceEffects("oil_painting")} >
                    Oil Painting
                </Button>

                <Button variant="contained" align="left" onClick={() => this.createAdvanceEffects("vibrance")} >
                    vibrance
                </Button>

                <Button variant="contained" align="left" onClick={() => this.resetFilters(["vignette", "cartoonify", "vibrance", "oil_paint"])} color="primary">
                    Reset
                </Button>
            
            </Box>
        </CardContent>
    </Card>
</Grid>

For each of these filters we have code that push transformation to our state which applied to image

createAdvanceEffects(type) {

    let transforms = this.state.transforms;

    switch(type) {
        case "cartoon":
            const transform = {
                key: "cartoonify",
                value: "20:60"
            }
            transforms = this.getUpdatedTransform(transforms, transform);
            break;
        case "vignette":
            const transform_v = {
                key: "vignette",
                value: "30"
            }
            transforms = this.getUpdatedTransform(transforms, transform_v);
            break;
        case "oil_painting":
            const transform_p = {
                key: "oil_paint",
                value: "40"
            }
            transforms = this.getUpdatedTransform(transforms, transform_p);
            break;
        case "vibrance":
            const transform_vb = {
                key: "vibrance",
                value: "70"
            }
            transforms = this.getUpdatedTransform(transforms, transform_vb);
            break;
        default:
            break;

    }

    this.setState({transforms});
    
}