Convert Deep Learning Models between PyTorch, TensorFlow, and MATLAB

Convert Deep Learning Models between PyTorch, TensorFlow, and MATLAB » Artificial Intelligence

In this blog post we are going to show you how to use the newest MATLAB functions to:

This is a brief blog post that points you to the right functions and other resources for converting deep learning models between MATLAB, PyTorch®, and TensorFlow™. Two good resources to get started with are the documentation topics Interoperability Between Deep Learning Toolbox, TensorFlow, PyTorch, and ONNX and Tips on Importing Models from TensorFlow, PyTorch, and ONNX.

If you have any questions about the functionality presented in this blog post or want to share the exciting projects for which you are using model conversion, comment below.

 

Import Models into MATLAB

You can import models from PyTorch or TensorFlow with just one line of code.

What?	Import PyTorch models.	Import TensorFlow models.	Import PyTorch and Tensorflow models interactively.
How?	Use the importNetworkFromPyTorch function.	Use the importNetworkFromTensorFlow function.	Use the Deep Network Designer app.
When?	Introduced in R2022b.	Introduced in R2023b.	Import capability introduced in R2023b.

Quick Example

This example shows you how to import an image classification model from PyTorch. The PyTorch model must be pretrained and traced.

Run the following code in Python to get and save the PyTorch model. Load the MnasNet pretrained image classification model from the TorchVision library.

import torch
from torchvision import models
model = models.mnasnet1_0(pretrained=True)

Trace the PyTorch model. For more information on how to trace a PyTorch model, go to Torch documentation: Tracing a function. Then, save the PyTorch model.

X = torch.rand(1,3,224,224)
traced_model = torch.jit.trace(model.forward,X)
traced_model.save("traced_mnasnet1_0.pt")

Now, go to MATLAB and import the model by using the importNetworkFromPyTorch function. Specify the name-value argument PyTorchInputSizes so that the import function automatically creates and adds the input layer for a batch of images.

net = importNetworkFromPyTorch("mnasnet1_0.pt",PyTorchInputSizes=[NaN,3,224,224])

net = 

  dlnetwork with properties:

         Layers: [153×1 nnet.cnn.layer.Layer]
    Connections: [162×2 table]
     Learnables: [210×3 table]
          State: [104×3 table]
     InputNames: {'InputLayer1'}
    OutputNames: {'aten__linear12'}
    Initialized: 1

  View summary with summary.

Read the image you want to classify. Resize the image to the input size of the network.

Im_og = imread("peacock.jpg");
InputSize = [224 224 3];
Im = imresize(Im_og,InputSize(1:2));

The inputs to MnasNet require further preprocessing. Rescale the image. Then, normalize the image by subtracting the training images mean and dividing by the training images standard deviation. For more information, see Input Data Preprocessing.

Im = rescale(Im,0,1);

meanIm = [0.485 0.456 0.406];
stdIm = [0.229 0.224 0.225];
Im = (Im - reshape(meanIm,[1 1 3]))./reshape(stdIm,[1 1 3]);

Convert the image to a dlarray object. Format the image with the dimensions "SSCB" (spatial, spatial, channel, batch).

Im_dlarray = dlarray(single(Im),"SSCB");

Classify the image and find the predicted label.

prob = predict(net,Im_dlarray);
[~,label_ind] = max(prob);

Display the image and classification result.

imshow(Im_og)
title(strcat("Classification Result: ",ClassNames(label_ind)),FontSize=16)

 

Importing a model from TensorFlow is quite similar to importing a model from PyTorch. Of course you need to use the importNetworkFromTensorFlow function instead. Note that your TensorFlow model must be in the SavedModel format and saved by using the following code in Python.

model.save("myModelTF")

For more examples, check out the reference pages of the importNetworkFromPyTorch and importNetworkFromTensorFlow functions, and the documentation page on Pretrained Networks from External Platforms.

 

Why Import Models into MATLAB?

In short, import models into MATLAB because an imported network is a MATLAB network. This means, you can perform all the following tasks with built-in tools. To learn more, see Deep Learning Toolbox.

 

 

Export Models from MATLAB

You can export models to TensorFlow directly. To export a model to PyTorch, you must first convert the model to the ONNX model format.

What?	Export models to TensorFlow.	Export models to PyTorch.
How?	Use the exportNetworkToTensorFlow function.	Export via ONNX by using the exportONNXNetwork function.
When?	Introduced in R2022b.	ONNX export introduced in R2018a.

 

Quick Example

Load the pretrained SqueezeNet convolutional network by using imagePretrainedNetwork. This function, which was introduced in R2024a, is the easiest (and recommended) way to load pretrained image classification networks.

[net,ClassNames] = imagePretrainedNetwork("squeezenet");

Export the network net to TensorFlow. The exportNetworkToTensorFlow function saves the TensorFlow model in the Python package myModel.

exportNetworkToTensorFlow(net,"myModel")

And you are done with exporting!

If you want to test model in Python, you can use an image available in MATLAB. Read the image you want to classify. Resize the image to the input size of the network.

Im = imread("peacock.jpg");
InputSize = net.Layers(1).InputSize;
Im = imresize(Im,InputSize(1:2));

Permute the 2-D image data from the MATLAB ordering (HWCN) to the TensorFlow ordering (NHWC), where H, W, and C are the height, width, and number of channels of the image, respectively, and N is the number of images. For more information, see Input Dimension Ordering. Save the image in a MAT file.

ImTF = permute(Im,[4,1,2,3]);
filename = "peacock.mat";
save(filename,"ImTF")

The code below shows you how to use the exported model to predict in Python or save it in the SavedModel format. First, load the exported TensorFlow model from the package myModel.

import myModel
model = myModel.load_model()

Save the exported model in the TensorFlow SavedModel format. Saving the model in SavedModel format is optional. You can perform deep learning workflows directly with the model.

model.save("myModelTF")

Classify the image with the exported model.

import scipy.io as sio
x = sio.loadmat("peacock.mat")
x = x["ImTF"]

import numpy as np
x_np = np.asarray(x, dtype=np.float32)

scores = model.predict(x_np)

Read more on exporting deep neural networks to TensorFlow in this blog post. For more examples, see the exportNetworkToTensorFlow and exportONNXNetwork reference pages and the documentation page on Exporting Deep Neural Networks.