Create Amazing Image Style Effects with only a few lines of code in Deep Learnin...

Create Amazing Image Style Effects with only a few lines of code in Deep Learning!

Learn to implement Neural Style Transfer by using CNN

Neural Style Transfer is an optimization technique in which an Original Image and a Style Image (like an artwork by a famous painter or a pattern) are blended together to get an Output Image that is in the design or pattern of the Style Image. In other words, it can also be called as Image Style Transfer.

In this article, we are going to apply the Deep Learning method of Convolutional Neural Networks (CNN) to create an image effect in which the design or style of one image can be applied onto another Image to see creative effects.

Main Image, Style Image and the Output Image

The above shared picture is the one that shows the implementation of Style Transfer of Images using CNN.

Overview —

For our experiment, we will be using the following ‘ Original Image ’ and ‘ Style Image ’ shared below.

Style Image

From the above shared images, the ‘ Original Image ’ is the one on which we will apply the design of the ‘ Style Image ’ to get the final main output. The detailed code for the entire program is available at the end of this article.

Initial Steps —

In the initial steps of the program code, we resize both the Original Image and the Style Image to a size of 512X512 . The output size will be (512,512,3) . Where 3 denoted that the image is an RGB or Color Image.

main_image=main_image.resize((512,512))
style_image=style_image.resize((512,512))main_array=np.asarray(main_image,dtype='float32')
main_array=np.expand_dims(main_array,axis=0)
style_array=np.asarray(style_image,dtype='float32')
style_array=np.expand_dims(style_array,axis=0)

The next step is to reshape the image to a 4D- Tensor of shape (1, 512, 512, 3) by using expand_dims . We then create a new variable ‘ final_image ’ which will be our final output image.

height=512
width=512
main_image=backend.variable(main_array)
style_image=backend.variable(style_array)
final_image=backend.placeholder((1,height,width,3))input_tensor=backend.concatenate([main_image,style_image,final_image],axis=0)

CNN Architecture —

In this method, we will be applying the pre-trained VGG16 CNN Model by utilizing the concept of Transfer Learning .

model=VGG16(input_tensor=input_tensor,weights='imagenet', include_top=False)

VGG16 Architecture (Source)

According to a source Johnson et al . , to extract the features for the main image content layer we should select block1_conv2 And for the style layers we need to choose block1_conv2, block2_conv2, block3_conv3, block4_conv3, block5_conv3 . These are the layers that can accurately extract the features from the two images.

layer_features=layers['block2_conv2']feature_layers = ['block1_conv2', 'block2_conv2',
                  'block3_conv3', 'block4_conv3',
                  'block5_conv3']

As given in the paper, this combination of layers chosen is working correctly to get the required style transfer. However, you can try different combinations to get more accurate Style Transfer.

Main Loss —

Once we have finalized on the CNN model, we can now define a main loss function. It is the distance between the main image and our output image .

def main_loss(content, combination):     
return backend.sum(backend.square(content-combination))

Style Loss —

The style loss is similar to the main loss as it is the distance between the style image and our output image .

def style_loss(style,combination):
    S=gram_matrix(style)
    C=gram_matrix(combination)
    channels=3
    size=height * width
    st=backend.sum(backend.square(S - C)) / (4. * (channels ** 2) * (size ** 2))
return st

Final Loss —

Finally, we will define another loss called as the final loss which will regularize the final image.

def final_loss(x):
    a=backend.square(x[:,:height-1,:width-1,:]-x[:,1:,:width-1,:])
    b = backend.square(x[:, :height-1, :width-1, :] - x[:, :height-1, 1:, :])
return backend.sum(backend.pow(a + b, 1.25))

Optimization —

Once we define the three losses, we can state the Style Transfer as an optimization problem in which we aim to minimize all the three losses define above which is collectively called as global loss .

def eval_loss_and_grads(x):     
    x = x.reshape((1, height, width, 3))     
    outs = f_outputs([x])     
    loss_value = outs[0]     
    grad_values = outs[1].flatten().astype('float64')     
return loss_value, grad_values

Evaluator —

After this, we define a class Evaluator() in which we collectively combine all the above defined functions and use it in the main iteration for Style Transfer.

class Evaluator(object):
def __init__(self):
        self.loss_value=None
        self.grads_values=None

def loss(self, x):
assert self.loss_value is None
        loss_value, grad_values = eval_loss_and_grads(x)
        self.loss_value = loss_value
        self.grad_values = grad_values
return self.loss_value    def grads(self, x):
assert self.loss_value is not None
        grad_values = np.copy(self.grad_values)
        self.loss_value = None
        self.grad_values = None
return grad_values

Results —

In this, we will use the Limited-memory BFGS which is an optimization algorithm to perform the Style Transfer for 10 iterations.

evaluator=Evaluator()
iterations = 10
for i in range(iterations):
    print('Start of iteration -', i)
    ta = time.time()
    x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
                           fprime=evaluator.grads, maxfun=20)
    print(min_val)
    tb = time.time()
    print('Iteration %d done in %d seconds' % (i, tb - ta))

On visualizing the results with the L-BFGS algorithm we get the following image.

Final Image

From this above image, we can see that the Style of the ‘ Style Image ’ has successfully been imposed on the ‘ Original Image ’. Thus, we have successfully implemented the Image Style Transfer using CNN.

I am sharing the link to my github repository where you can find the entire code available for your reference.

Other Examples —

Here are some more examples of Image Style Transfer which have been implemented using this program.

Conclusion —

From this article, we have been able to utilize Deep Learning and CNN in particular, to create amazing Style Transfer Effects. Try tweaking the hyper-parameters, optimizers and also other CNN architectures to get new and different results. Till then, Happy Machine Learning!