25

Working with 3D data —  fastai2

 3 years ago
source link: https://towardsdatascience.com/working-with-3d-data-fastai2-5e2baa09037e?gi=e53a422e6943
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

Working with 3D data — fastai2

Understanding fastai DataBlock and creating the building blocks to work with sequences of images for deep learning applications

mYr6fey.jpg!web

May 29 ·9min read

jUz6Nrr.jpg!web

Photo by Olav Ahrens Røtne on Unsplash .

Introduction

Working with 3D data or sequences of images is useful for a wide range of applications. Consider you are working with 3D medical data, video data, temporal sequences of satellite images, or even several slices of a larger image. If you are working on such kind of problem you need some additional considerations beyond a simple image-based problem. How to perform augmentations for 3D data? Should these transformations be applied image-wise or sequence-wise, or maybe both? How to shuffle the training data? Should the sequence order be preserved? The answers to these questions depend on the type of problem. It is, however, useful to have a framework able to adapt to all these different possibilities. fastai is modular and extremely versatile making it a very useful tool for Deep Learning practitioners as well as researchers.

In this story, I will start by covering a basic example of fastai DataBlock — the building blocks to create dataloaders. Then I will show how to code additional features to allow creating dataloaders with 3D data using the DataBlock, including augmentations for sequences of images.

A working example is provided on this Kaggle kernel .

fastai DataBlock

The code below shows an example of the fastai DataBlock class for a typical image-based dataset. If you are new to fastai, you can find several more examples in the fastai documentation .

dblock = DataBlock(blocks     = (ImageBlock, CategoryBlock),
get_items = get_image_files,
get_y = label_func,
splitter = RandomSplitter(),
item_tfms = Resize(224),
batch_tfms = aug_transforms())
  • blocks — this argument receives a tuple that defines the type of inputs and targets for the DataBlock . In this case, we have ImageBlock and CategoryBlock, therefore DataBlock will expect image inputs and categorical targets — as the names suggest.
  • get_items — this argument receives a function that returns a list of items. In this case, get_image_files simply returns the paths for all the image files is some directory and subdirectories.
  • get_y — this argument receives a function to define the labels. In this case the category of the corresponding item. get_x can also be used in some cases as we will see later. Keep in mind that get_x and get_y receive the output of get_items and define respectively which information ImageBlock and CategoryBlock will receive. In this example, get_x is not needed because get_image_files already returns the list paths to the images — as needed for the ImageBlock .
  • splitter — used to indicate how to split the data into train and validation sets.
  • item_tfms — used to define transformations applied to each image on CPU.
  • batch_tfms — used to define transformations applied to each batch on the GPU.

As you can see this structure is very flexible and works with virtually any kind of data as long as each argument is defined accordingly. For more advanced applications, it may be necessary to define custom block types and functions to create the batches. That is the case for working with 3D data. However, as I will show in the six steps below, this procedure is quite simple once you are familiar with the structure of DataBlock .

Step 0. Data source

I will use MNIST data for this example. I defined sequences of numbers from 0 to 9 to be our image sequences — this makes debugging easier. The table below shows how the dataframe is organized.

yi6ZbyY.png!web

Example of the dataframe. Image created by the author.

The columns in the dataframe are:

  • file : the path to each image;
  • sequence_id : unique identifier for each sequence;
  • sequence_order : the order of the elements in the sequence — in this case, corresponds to the digit from 0 to 9;
  • label : The target label for each sequence.

Step 1. DefiningImageSequenceBlock

To define an ImageSequenceBlock I used the following code:

class ImageSequence(Tuple):
@classmethod
def create(cls, image_files):
return cls(tuple(PILImage.create(f) for f in image_files))

def ImageSequenceBlock():
return TransformBlock(type_tfms = ImageSequence.create,
batch_tfms = int2float)
  • First, an ImageSequence is created. It is just a Tuple with all the images given in image_files list.
  • Then the ImageSequenceBlock is just a function that returns a fastai TransformBlock using ImageSequence . This is a simple adaptation from the fastai code for ImageBlock .

Step 2. Defining get items function

Notice that the create function in ImageSequence receives as arguments a list of image files. We need to create the function that will return such list and give it to get_items argument in DataBlock.

Based on the data source format — the dataframe in Step 0 — I defined the following class to create the list of sequences and labels:

class SequenceGetItems():
def __init__(self, filename_col, sequence_id_col, label_col):
self.fn = filename_col
self.seq = sequence_id_col
self.label = label_col

def __call__(self, df):
data = []
for fn in progress_bar(df[self.seq].unique()):
similar = df[self.seq] == fn
similar = df.loc[similar]
fns = similar[self.fn].tolist()
lbl = similar[self.label].values[0]
data.append([*fns, lbl])
return data
  • The attributes necessary for this class are just the column names on the dataframe corresponding to filenames, unique sequence identifiers and the labels.
  • When called, the __call__ method will receive the dataframe “df” as an argument and will return a list of lists. Each sublist consists of the image filenames for a sequence and the corresponding label in the last element of the sublist.

Now remember from the DataBlock example above that get_x and get_y receive the output from get_items and should separate what is the input and what is the target. In this case, they are as simple as this:

get_x = lambda t : t[:-1]
get_y = lambda t : t[-1]
  • As you see the get_x will select all elements of the list but the last that will instead be selected by get_y . This is just a consequence of how the sequences are created as discussed before — the label is simply the last element in each sublist.

Step 3. Put everything together in the DataBlock

With the code defined in the two steps above, building a DataBlock for our sequences of images is a straight-forward task:

dblock = DataBlock(
    blocks    = (ImageSequenceBlock, CategoryBlock),
    get_items = SequenceGetItems('file', 'sequence_id', 'label'), 
    get_x     = lambda t : t[:-1],
    get_y     = lambda t : t[-1],
    splitter  = RandomSplitter())
  • Notice that for the blocks argument we now give the ImageSequenceBlock defined in Step 1.
  • Then for the get_items argument, we use the SequenceGetItems class defined in Step 2.
  • get_x and get_y are very simple as discussed in the previous section.
  • As for the splitter, the usual fastai functions can be used. Another common option is IndexSplitter that allow to specify exactly which items are on the validation set.

I’m not including any item_tfms or batch_tfms yet but I will in a minute. However, let’s first look at an important additional piece — creating the dataloaders.

Step 4. Create the dataloaders

To create a dataloader from the DataBlock we will need to define a custom function to tell how the batch should be created. So far we just have sequences of images. What we want is a PyTorch tensor.

def create_batch(data):
xs, ys = [], []
for d in data:
xs.append(d[0])
ys.append(d[1])
xs = torch.cat([TensorImage(torch.cat([im[None] for im in x], dim=0))[None] for x in xs], dim=0)
ys = torch.cat([y[None] for y in ys], dim=0)
return TensorImage(xs), TensorCategory(ys)
  • This code concatenates the individual image tensors into a new tensor with an additional dimension and then the individual sequences together in a batch. If the images are 3 channel, 128x128 size, the length of the sequence is 10 and the batch-size is 8, then the tensor will have a shape of [8, 10, 3, 128, 128].
  • Notice that when I return the tensors I use TensorImage and TensorCategory . These are fastai types that provide very useful functionality.

Now, to create the dataloaders, the following line does the trick:

dls = dblock.dataloaders(df, bs=8, create_batch=create_batch)
  • Notice that df is our dataframe mentioned in Step 0, bs is the batch-size and a custom create_batch function is given.

Step 5. Visualizing the data

This step is not fundamental for the functionality but it is always good to be able to display the data and check if everything is working as intended!

def show_sequence_batch(max_n=4):
xb, yb = dls.one_batch()
fig, axes = plt.subplots(ncols=10, nrows=max_n, figsize=(12,6), dpi=120)
for i in range(max_n):
xs, ys = xb[i], yb[i]
for j, x in enumerate(xs):
axes[i,j].imshow(x.permute(1,2,0).cpu().numpy())
axes[i,j].set_title(ys.item())
axes[i,j].axis('off')

The show_sequence_batch function samples one batch from the dataloaders and produces a result similar to the following image where each row is a sequence:

byuEVjN.png!web

Result of show_sequence_batch function. Image created by the author.

Notice that the numbers above each image correspond to the sequence identifier — that’s why in this example they are the same in each row.

Step 6. Adding image and sequence augmentations

As our batches have an “extra” dimension — corresponding to the sequence length — we can’t apply the fastai image augmentations out-of-the-box. Furthermore, as discussed in the introduction of this story, in some problems we need to apply transformations sequence-wise — in order to preserve spatial coherence among the elements of the sequence that we may want to feed to a model with 3D convolutions.

The simplest way to achieve the desired result — or at least the simplest way I could think — is to notice that transformations are applied identically to each image channel. Therefore, if we play with the tensor shape — using PyTorch view method— we can apply the basic image transformations sequence-wise or image-wise. After the transformations, we can just reshape the tensor back to the original shape.

To this end, I defined two classes: SequenceTfms and BatchTfms . These two classes are defined as subclasses of fastai Transform . One aspect to notice about fastai Transform is that the encodes method is like the forward method in PyTorch modules or the usual __call__ method in Python classes.

class SequenceTfms(Transform):
def __init__(self, tfms):
self.tfms = tfms

def encodes(self, x:TensorImage):
bs, seq_len, ch, rs, cs = x.shape
x = x.view(bs, seq_len*ch, rs, cs)
x = compose_tfms(x, self.tfms)
x = x.view(bs, seq_len, ch, rs, cs)
return x

class BatchTfms(Transform):
def __init__(self, tfms):
self.tfms = tfms

def encodes(self, x:TensorImage):
bs, seq_len, ch, rs, cs = x.shape
x = x.view(bs*seq_len, ch, rs, cs)
x = compose_tfms(x, self.tfms)
x = x.view(bs, seq_len, ch, rs, cs)
return x
  • Both classes are initialized with a list of augmentations to be applied to the data.
  • SequenceTfms apply the transformations sequence-wise (same transformation for all images in the same sequence).
  • BatchTfms apply the transformations image-wise.

The complete DataBlock now looks like this:

affine_tfms, light_tfms = aug_transforms(flip_vert=True)
brightness = lambda x : x.brightness(p=0.75, max_lighting=0.9)
contrast   = lambda x : x.contrast(p=0.75, max_lighting=0.9)

dblock = DataBlock(
    blocks     = (ImageSequenceBlock, CategoryBlock),
    get_items  = SequenceGetItems('file', 'sequence_id', 'label'), 
    get_x      = lambda t : t[:-1],
    get_y      = lambda t : t[-1],
    splitter   = RandomSplitter(valid_pct=0.2, seed=2020),
    item_tfms  = Resize(128),
    batch_tfms = [SequenceTfms([affine_tfms]), 
                  BatchTfms([brightness, contrast])])

dls = dblock.dataloaders(df, bs=8, create_batch=create_batch)

Note: So far I’ve not been able to use the light_tfms given by aug_transforms using this method — that’s why I manually define the brightness and contrast transformations that are simply methods of ImageTensor fastai type.

The result of show_sequence_batch visualization is now the following:

f63mIfn.png!web

Result of show_sequence_batch function with image and sequence augmentations. Image created by the author.
  • Notice that in the first row the images are rotated but they are consistently rotated across the sequence as the affine transformations were applied sequence-wise.
  • For the brightness and contrast transformations notice that they are randomly applied over individual images.

As you can see, this framework can easily be customized according to the type of data/problem.

Conclusion

In this story, I covered how to work with 3D data for deep learning applications using fastai DataBlock . A working example of this code is provided on this Kaggle kernel .

It took me some time and several iterations to get to the code presented here but all this learning process gave me a better understanding of fastai DataBlock and how it can be so versatile!

Thanks for reading! Feel free to leave a comment if you have additional questions regarding this topic.


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK