This notebook uses the approach of generating bounding boxes from Class Activation Maps (CAM). CAM was introduced by Bolei Zhou et al. in the paper Learning Deep Features for Discriminative Localization and is a great tool for model interpretation.

Grad CAM is a variation of CAM that was introduced in Grad-CAM: Why Did You Say That? Visual Explanations from Deep Networks via Gradient-based Localization

Grad CAM is similar to CAM in that it is a weighted combination of feature maps but followed by a ReLU. The output of Grad CAM is a “class-discriminative localization map” where the hot part of the heatmap corresponds to a particular class.

You can view this notebook on github

CAM uses the output of the last convolutional layer to provide a heatmap visualization. A way of acccessing the activations during the training phase in Pytorch is done by using a hook. A hook is basically a function that is executed when either the forward or backward pass is called.

Fastai conveniently has a Hook class

class Hook():
    def __init__(self, m):
        self.hook = m.register_forward_hook(self.hook_func)   
    def hook_func(self, m, i, o): self.stored = o.detach().clone()
    def __enter__(self, *args): return self
    def __exit__(self, *args): self.hook.remove()

The gradients of every layer are calculated by PyTorch during the backward pass. To access the gradients you can register a hook on the backward pass and store these gradients.

Again conveniently fastai has the HookBwd class.

class HookBwd():
    def __init__(self, m):
        self.hook = m.register_backward_hook(self.hook_func)   
    def hook_func(self, m, gi, go): self.stored = go[0].detach().clone()
    def __enter__(self, *args): return self
    def __exit__(self, *args): self.hook.remove()

Load the Data and Create DataBlock

The data and DataBlock creation is can be found in this notebook

pneu = untar_data(URLs.SIIM_SMALL)
df = pd.read_csv(f'{pneu}/labels.csv')
p_items = get_dicom_files(f'{pneu}/train')

item_tfms = Resize(266)
batch_tfms = [RandomResizedCropGPU(226), *aug_transforms(do_flip=False, flip_vert=False, max_rotate=10.,
                                                         min_zoom=1., max_zoom=1.1, max_lighting=0.2, max_warp=0.1),
                                                         Normalize.from_stats(*imagenet_stats)]

splitter = dicom_splitter(p_items, valid_pct=0.2, seed=7)

pneumothorax = DataBlock(blocks=(ImageBlock(cls=PILDicom), CategoryBlock),
                   get_x=lambda x:pneu/f"{x[0]}",
                   get_y=lambda x:x[1],
                   splitter=splitter,
                   item_tfms = item_tfms,
                   batch_tfms = batch_tfms)

dls = pneumothorax.dataloaders(df.values, bs=32, num_workers=0)

learn = vision_learner(dls, 
                    'resnetblur50' ,
                     pretrained=True,
                    loss_func=LabelSmoothingCrossEntropyFlat(), 
                    metrics=accuracy, 
                    cbs=[ShowGraphCallback(), ReduceLROnPlateau(monitor='valid_loss', min_delta=0.1, patience=1),\
                         GradientClip, \
                         SaveModelCallback(monitor='accuracy',fname='siim_small_best',comp=np.greater, with_opt=True)])

200 50

learn.fit(3, 1e-3)

Better model found at epoch 0 with accuracy value: 0.6200000047683716.

Epoch 1: reducing lr to 0.0001
Epoch 2: reducing lr to 1e-05

Generating the heatmaps

This is the walk through of how the heatmaps are generated. First we grab an image.

test_path = 'C:/Users/avird/.fastai/data/siim_small/test'
test_files = get_dicom_files(test_path)
test_one = test_files[0]

We can use learn.predict to predict the class of the test image. This returns the class, prediction and probabilities

cl, pred, probs = learn.predict(test_one)
print(cl, pred, probs)

No Pneumothorax TensorBase(0) TensorBase([0.9962, 0.0038])

To be able to generate the correct heatmap for the predicted class Pneumothorax we need to pass the predicted tensor value which is 1 when generating the heatmap. Recall that the number of classes in a dataset can be found by doing:

dls.vocab

['No Pneumothorax', 'Pneumothorax']

Where No Pneumothorax is class 0 and Pneumothorax is class 1

You can define the predicted tensor like so:

cls = pred.item()
cls

0

first is handy fastcore function that takes the first item in a list. In this case we can create a list comprehension of files in the test folder and x will be the first item in that folder

x, = first(dls.test_dl([file for file in test_files]))

Generate the image

x_dec = TensorImage(dls.train.decode((x,))[0][0])
x_dec.show();

Grab the gradients using Hook and HookBwd and here we are grabbing them from the last layer learn.model[0][-1]

with HookBwd(learn.model[0].model.layer4[-1]) as hookg:
    with Hook(learn.model[0].model.layer4[-1]) as hook:
        output = learn.model.eval()(x.cuda())
        act = hook.stored
    output[0, cls].backward()
    grad = hookg.stored

Generate the heatmap

w = grad[0].mean(dim=[1,2], keepdim=True)
cam_map = (w * act[0]).sum(0)

Check the shape of cam_map

cam_map.shape

torch.Size([8, 8])

In this case the heatmap is a 8 by 8 square. Note that the shape of cam_map is determined by the item_tfms shape set in the DataBlock. The larger the image used during training the larger the heat map.

We can view the heatmap using show_image

show_image(cam_map, figsize=(4,4));

The location on the heatmap that has the highest activations is the bright yellow square at position (4, 7).

We can view the heatmap on the image like so:

_,ax = plt.subplots()
x_dec.show(ctx=ax)
ax.imshow(cam_map.detach().cpu(), alpha=0.6, extent=(0, x_dec.shape[1], x_dec.shape[2],0),
              interpolation='bilinear', cmap='magma');

How do you generate the bounding boxes?

Now we need to be able to generate a bounding box around the area with the highest activation.

To do this we need to iterate over the cam_map values and get the indexes of the x and and y locations where the activations are the highest.

cam_map is a 8 by 8 TensorDicom

cam_map, cam_map.shape

(TensorDicom([[-0.0323, -0.0516, -0.0334, -0.0313, -0.0308, -0.0185, -0.0270, -0.0374],
         [-0.0312,  0.0720,  0.1065,  0.0011,  0.0067, -0.0115,  0.0088, -0.0220],
         [ 0.0602,  0.2496,  0.3080,  0.2261,  0.0381,  0.0546,  0.0313, -0.0036],
         [-0.0693,  0.0633,  0.1196,  0.1218,  0.0321,  0.0277,  0.0128,  0.0244],
         [-0.0234,  0.0895,  0.1049,  0.0646,  0.0707,  0.0333,  0.0303,  0.0101],
         [ 0.0038,  0.0531,  0.0158,  0.0226,  0.0879,  0.0443,  0.0624,  0.0166],
         [-0.0166, -0.0376, -0.0312,  0.0028,  0.0026,  0.0564, -0.0378, -0.0465],
         [ 0.0076, -0.0125, -0.0063, -0.0015, -0.0026,  0.0116, -0.0725, -0.0699]],
        device='cuda:0'),
 torch.Size([8, 8]))

Convert the cam_map into a tensor but first we need to copy the tensor to host memory first by using .cpu()

arr = np.array(cam_map.cpu())
ten_map = tensor(arr)

Find the value of the tensor with the maximum value

ten_map.max()

tensor(0.3080)

Now to get the indexes of the maximum tensor we start by first finding the maximum value in each of the rows in ten_map

val = []
for i in range(0, ten_map.shape[0]):
    index, value = max(enumerate(ten_map[i]), key=operator.itemgetter(1))
    val.append(value)

val

[tensor(-0.0185),
 tensor(0.1065),
 tensor(0.3080),
 tensor(0.1218),
 tensor(0.1049),
 tensor(0.0879),
 tensor(0.0564),
 tensor(0.0116)]

We now have the maximum value of each row. Now to get the y_index and confirm that it matches the maximum value

y_index, y_value = max(enumerate(val), key=operator.itemgetter(1))
print(y_index, y_value)

2 tensor(0.3080)

The highest value of 0.1026 is found at index 4(remember that indexes start from 0 so index 4 is the 5th row)

Get the x_index

x_index, x_value = max(enumerate(ten_map[y_index]), key=operator.itemgetter(1))
x_index, x_value

(2, tensor(0.3080))

We can confirm that the highest value (0.1026) can be found at index [4,7]

Now we need to match that index to the image

To do this we need to know the shape of the image

imz = pydicom.dcmread(test_files[0]).pixel_array 
imz.shape

(1024, 1024)

And since the heatmap is broken down into 8 by 8 squares we need to divide the image into 8 by 8 sections so that they correspond to each section of the heatmap. In this case the image is already a square but in reality it is not usually the case.

cms = cam_map.shape[0]
cms

8

Along the x-axis

x_ = imz.shape[1] // cms
x_

128

Along the y-axis

y_ = imz.shape[0] // cms
y_

128

So the image now has 8 by 8 squares each of size 128 by 128.

To match the square with highest activations we multiply the x and y indexes we calculated earlier to the size of each square

x = x_index * x_
x

256

y = y_index * y_
y

256

Using TensorBBox we can now plot a bounding box around the area of the image which has the highest activations

box = TensorBBox([x,y, (x + (imz.shape[0]//cms)),(y + (imz.shape[1]//cms))])
_,ax = plt.subplots()
x_dec.show(ctx=ax)
ax.imshow(cam_map.detach().cpu(), alpha=0.6, extent=(0, x_dec.shape[1], x_dec.shape[2],0),
              interpolation='bilinear', cmap='magma');
ctx = show_image(imz)
box.show(ctx=ctx);

get_maps

get_cmaps is a handy fmi function from where you can easily view activation maps

We first have to define which layer in the model you want to view. For this case I want to view the last layer

layer = learn.model[0].model.layer4[-1]

get_cmaps allows for some flexibility:

sanity = if you want to check actual values
show_maps = display the class activation map
show_cmap = display the test image with the class activation map super-imposed

get_cmaps(test_one, dls, learn, layer=layer, sanity=True, show_maps=True, show_cmap=True)

No Pneumothorax
TensorBase([0, 1])
TensorBase([0.9962, 0.0038])

The predicted class for this image is Pneumothorax which is tensor 1 with a probability of 0.58.

get_cmaps displays the activation maps of each class (predicted first) and also displays the image with super imposed activation maps

get_boxes

get_boxes creates bounding boxes at locations with the highest activations in each class activation map.

It also includes code updates to TensorBBox and LabelledBBox classes so that you can change the color of the bounding boxes

get_boxes(test_one, dls, learn, layer=layer, sanity=False , show_maps=False, show_img=True, color='red')

About

fmi is a library built on top of fastai and adds additional functionality to fastai's medical imaging module. This project is still a work in progress.

View the library on Github

Installation

git clone https://github.com/asvcode/fmi.git

Tutorials

Here is a list of tutorials to get you stated with the fmi library

Create Bounding Boxes from Class Activation Maps - New

Medical Imaging Tutorial - Updated

Explore & Preprocessing Tutorial

Dicom Splitter Examples

Signal Processing Tutorial - Not Complete

A Matter of Grayscale

DICOM images typically contain between 12–16 bits/pixel, which corresponds to approximately 4,096 to 65,536 shades of gray^[1]. Most medical displays and regular computer screens are often limited to 8 bits or 256 shades of gray. There are high end medical displays that can display 1024 shades of gray (like the ones optimized for mammography).

However even with a computer screen that can display 256 shades of gray our eyes can typically only detect a 6% change in grayscale^[2]

That means we can typically only detect about $100/6$ = 17 shades of gray

The [Hounsfield unit]^[3] (HU) scale is a quantitative scale for describing radiodensity. It applies a linear transformation where the radiodensity of distilled water at standard pressure and temperature (STP) is defined as 0 HU, while the radiodensity of air at STP is defined as -1000 HU.

(Image credit^[4]) Most images will require viewing between -1000 HU (which is a reference for air) and +1000 HU (which typically references hard bone).

So a DICOM image can have a range of 2000 HU (from -1000 HU to +1000 HU) and if we wanted to display this range on a computer screen which can only display 256 shades of grey: $2000/256 = 8$. Then each shade of gray would have a difference of 8 HU.

The human eye can only detect a 6% change in grayscale so for humans to detect a difference in densities (within the image range of 2000 HU), each variation has to vary by: $256/17 * 8 = 120$ HU. The difference between normal and pathologically altered tissue is usually a lot less than 120 HU and this is why applying windows is important.

Windowing

Windowing can be thought of as a means of manipulating pixel values in order to change the apperance of an image so particular structures within the image are highlighted.

To futher explain how windowing works we use this dataset, import the fastai libraries and load a test image.

View the pixel_array

show_image(patient6.pixel_array, cmap=plt.cm.bone);

The difference when using fastai is that by default it will display the normalized image when the show function is used.

normalized = patient6.show(); normalized;

1 frame per file

Clearly there is alot more depth displayed in the image. However this can be an issue when trying to localize areas that are normal to those that have been pathologically altered due to a condition. In most cases the difference in Hounsfield units between normal and pathologically altered tissue can be very small.

The normalized image displays a wide range of tissue densities(ranging from -1000HU (air) to around +1000HU (bone)). As mentioned above a regular computer screen can only display 256 shades of grey and our eye can only deduct about a 6% change in grayscale^[2]

The basis of applying windows is to focus down the 256 shades of grey into a narrow region of Hounsfled units that contain the relevant densities of tissues we may be interested in.

`Fastai`'s in-built windows

The fastai medical imaging library conveniently provides a number of window ranges that can be used by its windowed function. The windows can be called by using dicom_windows

dicom_windows

namespace(abdomen_soft=(400, 50), brain=(80, 40), brain_bone=(2800, 600), brain_soft=(375, 40), liver=(150, 30), lungs=(1500, -600), mediastinum=(350, 50), spine_bone=(1800, 400), spine_soft=(250, 50), stroke=(8, 32), subdural=(254, 100))

How does windowing work?

A window has 2 values:

window level or center, also known as brightness, l
window width or range, also known as contrast, w

Window pixel values are calculated using the following formulas:

lowest_visible_value = window_level - window_width / 2
highest_visible_value = window_level + window_width / 2

Using the brain window as an example: it has a window width of 80 and a window level of 40.

lowest_visible_value = 40 - (80/2) = 0
highest_visible_value = 40 + (80/2) = 80

In this case the lowest_visible_value will be 0 and the highest_visible_value of 80. This means that every pixel value greater than 80 will show up as white and any value below 0 will show up as black.

To see what this looks like at the pixel level we can scale down to a small portion of the image and see what effect windowing has.

scaled = scaled_px(patient6)
crop = scaled[280:400,300:450]
show_image(crop, cmap='bone');

Scale down even further (picture above is to illustrate the area of the image we are looking at)

cropped = scaled[340:385,300:350]
show_image(cropped, cmap='bone');

We can now plot each pixel value

df = pd.DataFrame(cropped)
#uncomment below to view locally
#df.style.set_properties(**{'font-size':'8pt'}).background_gradient('bone').format("{:.1f}")

Using the lung window which has a window width of 1500 and a window level of -600.

lowest_visible_value = -600 - (1500/2) = -1350
highest_visible_value = -600 + (1500/2) = +150

In this case the lowest_visible_value will be -1350 and the highest_visible_value of +150. This means that every pixel value greater than +150 will show up as white and any value below -1350 will show up as black.

windowed_image = cropped.windowed(*dicom_windows.lungs)

show_image(windowed_image, cmap=plt.cm.bone);

dfs = pd.DataFrame(windowed_image)
#uncomment below to view locally
#dfs.style.set_properties(**{'font-size':'6pt'}).background_gradient('bone').format("{:.1f}")

The pixel intensities have been scaled to values within the window range. To check that the window range is working as planned we can compare the pixel values of the original and the windowed image. Any pixel value above +150 will show up as white and any pixel value below -1350 will show up as black with various shades between those values.

#original image
df[1:2].style.set_properties(**{'font-size':'8pt'}).format("{:.1f}")

#windowed image
dfs[1:2].style.set_properties(**{'font-size':'8pt'}).format("{:.1f}")

Starting with the first pixel on the left, pixel value of 234 is higher than the highest_visible_value of +150 so this pixel will show up as white on the windowed image (represented by a 1.0). The pixel value at column 21 is -731 which is not below the lowest_visible_value of -1350 so will not be shown as black but is represented by a shaded color(depending on the cmap setting) with a value of 0.4.

Significance of changing window settings

Window Width

Increasing the window width will decrease the contrast of the image
Decreasing the window width will increase the contrast of the image

high_width = patient6.windowed(w=1000, l=40)
low_width = patient6.windowed(w=100, l=40)

show_images([high_width, low_width], titles=['high window width', 'low window width'], figsize=(7,7))

Window Level

Increasing the window level will decrease the brightness of the image
Decreasing the window level will increase the brightness of the image

high_level = patient6.windowed(w=200, l=200)
low_level = patient6.windowed(w=200, l=-200)

show_images([high_level, low_level], titles=['high window level', 'low window level'], figsize=(7,7))

Various window ranges

There are numerous window ranges as specified in dicom_windows or you could create your own by specifying the window width and window center. These are the most common window ranges used when particularly looking at images of the lungs.

Lung Window

Window settings: (W:1600, L:-600)
Usually used with a wide window to provide good resolution and to visualize a wide variety of densities.

lung = plt.imshow(patient6.windowed(*dicom_windows.lungs), cmap=plt.cm.bone); lung;

Mediastinum Window

Window settings: (W:500, L:50)
The mediastinum is the area that separates the lungs. It is surrounded by the breastbone in front and the spine in back, with the lungs on either side. It encompasses the heart, aorta, esophagus, thymus (a gland in the back of the neck) and trachea (windpipe).

mediastinum = plt.imshow(patient6.windowed(*dicom_windows.mediastinum), cmap=plt.cm.bone); mediastinum;

PE Window

Window settings: (W:700, L:100) - this is not part of dicom_windows but you can easily use custom window ranges by specifying the window level and window width
Window range specifically used for looking at pulmonary embolisms

pe = plt.imshow(patient6.windowed(l=100, w=700), cmap=plt.cm.bone); pe;

Each window highlights particular ranges that make it easier for a radiologist to see if there are any changes between normal and pathologically altered tissue. If we compare the 3 images above with a non-windowed image we can clearly see why windowing is important.

show_images([patient6.hist_scaled(), patient6.windowed(*dicom_windows.lungs),patient6.windowed(*dicom_windows.mediastinum),\
             patient6.windowed(l=100, w=700)], titles=['normalized', 'lung', 'mediastinum', 'pe'], figsize=(17,17));

References

¹Increasing the Number of Gray Shades in Medical Display Systems—How Much is Enough?

²Understanding CT windows, levels and densities

³Hounsfield scale

⁴Image Credit

Note: Functions listed with a ✔ are custom functions

Goal:

The goals of this notebook are to:

look through how to select specific masks for an image
how to get the selective mask ready for the DataBlock
based on the dataset from this competition: Prostate cANcer graDe Assessment (PANDA) Challenge and you can view my kaggle kernel

This notebook does not use DICOMS

For example:

Original Mask

Selective Mask

Motivation

There more than 1 million new prostate cancer diagnoses reported every year, in fact prostate cancer is the second most common cancer among males worldwide that results in more than 350,000 deaths annually. Diagnosis of prostate cancer is based on the grading of prostate tissue biopsies. These tissue samples are examined by a pathologist and scored according to the Gleason grading system.

The grading process consists of finding and classifying cancer tissue into so-called Gleason patterns (3, 4, or 5) based on the architectural growth patterns of the tumor

(picture: courtesy: Kaggle)

After the biopsy is assigned a Gleason score, it is converted into an ISUP grade on a 1-5 scale

Dataset

The dataset consists of about 10,600 images and masks

#Load the dependancies
from fastai.basics import *
from fastai.callback.all import *
from fastai.vision.all import *

import seaborn as sns
import numpy as np
import pandas as pd
import os
import cv2

sns.set(style="whitegrid")
sns.set_context("paper")

matplotlib.rcParams['image.cmap'] = 'ocean_r'

source = Path("D:/Datasets/prostate_png")
files = os.listdir(source)
train = source/'train_images'
mask = source/'train_label_masks'
train_labels = pd.read_csv(source/'train.csv')

def view_image(folder, fn):
    if folder == train:
        filename = f'{folder}/{fn}.png'
    if folder == mask:
        filename = f'{folder}/{fn}_mask.png'
    file = Image.open(filename)
    t = tensor(file)
    if folder == train:
        show_image(t)
    if folder == mask:
        show_image(t[:,:,2])

Lets view an image

view_image(train, '0005f7aaab2800f6170c399693a96917')

And the corresponding mask

view_image(mask, '0005f7aaab2800f6170c399693a96917')

The dataset is categorized by both isup_grade and gleason_score. What is noticed is that the masks have different intensities. For example we can specify a function that will display the image, the mask and plot a histogram of the intensites.

def view_images(file, mask, fn):
    ima = f'{file}/{fn}.png'
    msk = f'{mask}/{fn}_mask.png'
    ima_file = Image.open(ima); ima_t = tensor(ima_file)
    ima_msk = Image.open(msk); msk_t = tensor(ima_msk)
    
    fig, (ax1, ax2, ax3) = plt.subplots(1,3, figsize = (20, 6))
    s1 = show_image(ima_t, ax=ax1, title='image')
    s2 = show_image(msk_t[:,:,2], ax=ax2, title='mask')
    s3 = plt.hist(msk_t.flatten()); plt.title('mask histogram')
    plt.show()

view_images(train, mask, '06636cdd43041e78141f2f5069fa62d5')

Plotting a histogram of the mask intensities shows that the bulk of the intensity is between 0 and 1 and this corresponds to the the bulk of the pixels which is the outline of the mask (light blue)

Here are some more examples

view_images(train, mask, '0d3159cd1b2495cc82637ececf63ed41')

view_images(train, mask, '08134913a9aa1d541f719e9f356f9378')

Can see that the bulk of the pixels within the mask are the light blue areas of the mask which correspond to the the out outline of the mask itself.

Selective Mask ✔

To be able to view the mask images at different intensities I adapted a function from fastai's medical imaging library (which is typically geared towards working with DICOM images).

Load the medical imaging library

from fastai.medical.imaging import *

This library has a show function that has the capability of specifying max and min pixel values so you can specify the range of pixels you want to view within an image (useful when DICOM images can vary in pixel values between the range of -32768 to 32768).

You can easily adapt any function in fastai2 using @patch and it just works! In this case I am adapting the show function so you can specify min and max pixel values for this dataset.

@patch
@delegates(show_image)
def show(self:PILImage, scale=True, cmap=plt.cm.ocean_r, min_px=None, max_px=None, **kwargs):
    px = tensor(self)
    if min_px is not None: px[px<min_px] = float(min_px)
    if max_px is not None: px[px>max_px] = float(max_px)
    show_image(px, cmap=cmap, **kwargs)

We will also have to define another function that will allow us to view the selective masks

def selective_mask(file, mask, fn, min_px=None, max_px=None):
    ima = f'{file}/{fn}.png'
    msk = f'{mask}/{fn}_mask.png'
    ima_file = Image.open(ima); ima_t = tensor(ima_file)
    ima_msk = Image.open(msk); msk_t = tensor(ima_msk)
    msk_pil = PILImage.create(msk_t[:,:,2])
    
    fig, (ax1, ax2, ax3, ax4) = plt.subplots(1,4, figsize = (20, 6))
    s1 = show_image(ima_t, ax=ax1, title='image')
    s2 = show_image(msk_t[:,:,2], ax=ax2, title='mask')
    s3 = msk_pil.show(min_px=min_px, max_px=max_px, ax=ax3, title=f'selective mask: min_px:{min_px}')
    s4 = plt.hist(msk_t.flatten()); plt.title('mask histogram')
    plt.show()

The plot shows the original image, the mask, the selective mask(in this case all intensities are shown hence the reason it looks the same as the mask image) and the histogram of intensities (again the bulk of pixels are within 0 and 1

selective_mask(train, mask, '08134913a9aa1d541f719e9f356f9378', min_px=None, max_px=None)

How about intensities above 1 (so getting rid of the bulk of pixels)

selective_mask(train, mask, '08134913a9aa1d541f719e9f356f9378', min_px=1, max_px=None)

Intensities above 2

selective_mask(train, mask, '08134913a9aa1d541f719e9f356f9378', min_px=2, max_px=None)

Intensities above 3

selective_mask(train, mask, '08134913a9aa1d541f719e9f356f9378', min_px=3, max_px=None)

The histogram does show some pixels above 4 but not many

selective_mask(train, mask, '08134913a9aa1d541f719e9f356f9378', min_px=4, max_px=None)

Looking at the selective masks side by side

msk = f'{mask}/08134913a9aa1d541f719e9f356f9378_mask.png'
ima_msk = Image.open(msk); msk_t = tensor(ima_msk)
msk_pil = PILImage.create(msk_t[:,:,2])

fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(1,5, figsize = (20, 6))
s1 = msk_pil.show(min_px=None, max_px=None, ax=ax1, title='original mask')
s2 = msk_pil.show(min_px=1, max_px=2, ax=ax2, title='1 and 2')
s3 = msk_pil.show(min_px=2, max_px=3, ax=ax3, title='2 and 3')
s4 = msk_pil.show(min_px=3, max_px=4, ax=ax4, title='3 and 4')
s4 = msk_pil.show(min_px=4, max_px=5, ax=ax5, title='4 and 5')
plt.show()

Comparing masks from different isup grades

Lets check an example with isup_grade of 0

isup_0 = train_labels[train_labels.isup_grade == 0]
isup_0[:1]

selective_mask(train, mask, '0005f7aaab2800f6170c399693a96917', min_px=None, max_px=None)

What about images with isup_grade of 5

isup_5 = train_labels[train_labels.isup_grade == 5]
isup_5[:1]

selective_mask(train, mask, '00928370e2dfeb8a507667ef1d4efcbb', min_px=None, max_px=None)

It looks like:

each mask has intensities based on its isup_grade

DataBlock

Lets see what the dataBlock would look like. fastai provides a very convenient way of getting the dataset ready for training for example you can specify blocks and getters where blocks can be images, labels etc and getters are where are the images or labels located.

For this we would have to create 2 custom functions, one so that that the dataloader can correctly view the images (as they are in .tiff format and fastai does not have an out of the box method of parsing these files and the second so that we only view the masks in the red channel

As we'll be using the csv file to load the images

def custom_img(fn):
    fn = f'{train}/{fn.image_id}.png'
    file = Image.open(fn)
    t = tensor(file)
    img_pil = PILImage.create(t)
    return img_pil

For the masks

We have to make a custom function show_selective so that we can pass the different intensities to the dataloader

def show_selective(p, scale=True, cmap=plt.cm.ocean_r, min_px=None, max_px=None):
    px = tensor(p)
    if min_px is not None: px[px<min_px] = float(min_px)
    if max_px is not None: px[px>max_px] = float(max_px)
    return px

def custom_selective_msk(fn):
    fn = f'{mask}/{fn.image_id}_mask.png'
    file = Image.open(fn)
    t = tensor(file)[:,:,2]
    ts = show_selective(t, min_px=None, max_px=None)
    return ts

Specify the blocks

Lets look at images and masks side by side just to see what the images and masks look like

Using Original Mask

blocks = (ImageBlock,
          ImageBlock)

getters = [
           custom_img,
           custom_selective_msk
          ]

prostate = DataBlock(blocks=blocks,
                 getters=getters,
                 item_tfms=Resize(128))

j = prostate.dataloaders(train_labels, bs=16)
j.show_batch(max_n=12, nrows=2, ncols=6)

The batch above shows the images and masks side by side, in this case the full mask is being shown. However what if we specify the mask intensites, as an example I want to get rid of the bulk of images which is predominantly the outline of the mask or any intensitiy below 1

Using Selective Mask

Specify the min_px value as 1 in the custom_selective_msk function:

def custom_selective_msk(fn):
    fn = f'{mask}/{fn.image_id}_mask.png'
    file = Image.open(fn)
    t = tensor(file)[:,:,2]
    ts = show_selective(t, min_px=1, max_px=None)
    return ts

blocks = (ImageBlock,
          ImageBlock)

getters = [
           custom_img,
           custom_selective_msk
          ]

prostate = DataBlock(blocks=blocks,
                 getters=getters,
                 item_tfms=Resize(128))

j = prostate.dataloaders(train_labels, bs=16)
j.show_batch(max_n=12, nrows=2, ncols=6)

We can see that the mask images are selective. The mask images that are fully purple correspond to images that have an isup_grade of 0

Now lets look at the images and masks over-layed on each other. For this we simply change the ImageBlock we used above for the mask images into a MaskBlock and lets add a CategoryBlock so that we can identify the isup_grade

def custom_selective_msk(fn):
    fn = f'{mask}/{fn.image_id}_mask.png'
    file = Image.open(fn)
    t = tensor(file)[:,:,2]
    ts = show_selective(t, min_px=None, max_px=None)
    return ts

blocks = (ImageBlock,
          MaskBlock,
          CategoryBlock)

getters = [
           custom_img,
           custom_selective_msk,
           ColReader('isup_grade')
          ]

prostate = DataBlock(blocks=blocks,
                 getters=getters,
                 item_tfms=Resize(224))

j = prostate.dataloaders(train_labels, bs=16)
j.show_batch(max_n=4)

Next Steps

There are still alot of experimentation to be done.

for example the data is extremely unbalanced but images with say isup_grade of 5 also have masks for grades 2,3 and 4
extract the portions within the masks to create a seperate dataset

DICOM images allows for various relationships between the pixel data and intended interpretation of how the image is displayed. The values are found in tag (0028,0004) which lists the Photometric Interpretation of the pixel data.

By using the easily accessible SLIM_SIIM dataset from fastai's list of databases, one can explore the head of the dicom file which contains a list of tags pertaining to various aspects of the data.

Monochrome

pneumothorax_source = untar_data(URLs.SIIM_SMALL)
items = get_dicom_files(pneumothorax_source, recurse=True, folders='sm')
patient_one = dcmread(items[0])
patient_one

(0008, 0005) Specific Character Set              CS: 'ISO_IR 100'
(0008, 0016) SOP Class UID                       UI: Secondary Capture Image Storage
(0008, 0018) SOP Instance UID                    UI: 1.2.276.0.7230010.3.1.4.8323329.6904.1517875201.850819
(0008, 0020) Study Date                          DA: '19010101'
(0008, 0030) Study Time                          TM: '000000.00'
(0008, 0050) Accession Number                    SH: ''
(0008, 0060) Modality                            CS: 'CR'
(0008, 0064) Conversion Type                     CS: 'WSD'
(0008, 0090) Referring Physician's Name          PN: ''
(0008, 103e) Series Description                  LO: 'view: PA'
(0010, 0010) Patient's Name                      PN: '16d7f894-55d7-4d95-8957-d18987f0e981'
(0010, 0020) Patient ID                          LO: '16d7f894-55d7-4d95-8957-d18987f0e981'
(0010, 0030) Patient's Birth Date                DA: ''
(0010, 0040) Patient's Sex                       CS: 'M'
(0010, 1010) Patient's Age                       AS: '62'
(0018, 0015) Body Part Examined                  CS: 'CHEST'
(0018, 5101) View Position                       CS: 'PA'
(0020, 000d) Study Instance UID                  UI: 1.2.276.0.7230010.3.1.2.8323329.6904.1517875201.850818
(0020, 000e) Series Instance UID                 UI: 1.2.276.0.7230010.3.1.3.8323329.6904.1517875201.850817
(0020, 0010) Study ID                            SH: ''
(0020, 0011) Series Number                       IS: "1"
(0020, 0013) Instance Number                     IS: "1"
(0020, 0020) Patient Orientation                 CS: ''
(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0010) Rows                                US: 1024
(0028, 0011) Columns                             US: 1024
(0028, 0030) Pixel Spacing                       DS: [0.168, 0.168]
(0028, 0100) Bits Allocated                      US: 8
(0028, 0101) Bits Stored                         US: 8
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 2110) Lossy Image Compression             CS: '01'
(0028, 2114) Lossy Image Compression Method      CS: 'ISO_10918_1'
(7fe0, 0010) Pixel Data                          OB: Array of 126284 elements

You can specifically call any of the tags, for this dataset the Photometric Interpretation is MONOCHROME2 where the pixel data is represented as a single monochrome image plane.

patient_one.PhotometricInterpretation

'MONOCHROME2'

According to the DICOM Standards Committee this interpretation can only be used when Samples per Pixel (0028, 0002) has a value of 1. We can confirm this:

patient_one.SamplesPerPixel

1

patient_one.show()

1 frame per file

More information about the different types of interpretations can be found here

YBR_FULL_422

Looking at another dataset, this time from the SIIM-ISIC Melanoma Classification competition.

msource = Path('D:/Datasets/Melanoma/test')
mitems = get_dicom_files(msource)
patient_two = dcmread(mitems[0])

patient_two

(0008, 0008) Image Type                          CS: ['DERIVED', 'SECONDARY']
(0008, 0014) Instance Creator UID                UI: 1.3.6.1.4.1.5962.99.3
(0008, 0016) SOP Class UID                       UI: "1.2.840.10008.5.1.4.1.1.77.1.4"
(0008, 0018) SOP Instance UID                    UI: 1.3.6.1.4.1.5962.99.1.7155.4713.1589853456601.1.1.0.0.0
(0008, 0020) Study Date                          DA: '20200519'
(0008, 0023) Content Date                        DA: '20200519'
(0008, 0030) Study Time                          TM: '015736'
(0008, 0033) Content Time                        TM: '015736'
(0008, 0050) Accession Number                    SH: ''
(0008, 0060) Modality                            CS: '"XC"'
(0008, 0070) Manufacturer                        LO: ''
(0008, 0080) Institution Name                    LO: 'ISDIS'
(0008, 0090) Referring Physician's Name          PN: ''
(0008, 1030) Study Description                   LO: 'ISIC 2020 Grand Challenge image'
(0008, 2218)  Anatomic Region Sequence   1 item(s) ---- 
   (0008, 0100) Code Value                          SH: '39937001'
   (0008, 0102) Coding Scheme Designator            SH: 'SCT'
   (0008, 0104) Code Meaning                        LO: 'Skin'
   ---------
(0010, 0010) Patient's Name                      PN: '"ISIC^0052060"'
(0010, 0020) Patient ID                          LO: '"ISIC_0052060"'
(0010, 0030) Patient's Birth Date                DA: ''
(0010, 0040) Patient's Sex                       CS: 'M'
(0010, 1010) Patient's Age                       AS: '065Y'
(0018, 0015) Body Part Examined                  CS: 'SKIN'
(0020, 000d) Study Instance UID                  UI: 1.3.6.1.4.1.5962.99.1.7155.4713.1589853456601.1.2.0
(0020, 000e) Series Instance UID                 UI: 1.3.6.1.4.1.5962.99.1.7155.4713.1589853456601.1.3.0.0
(0020, 0010) Study ID                            SH: 'ISIC_0052060'
(0020, 0011) Series Number                       IS: '"1"'
(0020, 0013) Instance Number                     IS: '"1"'
(0020, 0020) Patient Orientation                 CS: ''
(0028, 0002) Samples per Pixel                   US: 3
(0028, 0004) Photometric Interpretation          CS: 'YBR_FULL_422'
(0028, 0006) Planar Configuration                US: 0
(0028, 0010) Rows                                US: 4000
(0028, 0011) Columns                             US: 6000
(0028, 0100) Bits Allocated                      US: 8
(0028, 0101) Bits Stored                         US: 8
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 0301) Burned In Annotation                CS: 'YES'
(0028, 2110) Lossy Image Compression             CS: '01'
(0040, 0555)  Acquisition Context Sequence   0 item(s) ---- 
(7fe0, 0010) Pixel Data                          OB: Array of 2513486 elements

patient_two.PhotometricInterpretation

'YBR_FULL_422'

patient_two.SamplesPerPixel

3

In this case the interpretation is YBR_FULL_422 and the pixel data represents a color image described by one luminance (Y) and two chrominance planes (CB and CR). CB and CR values are sampled horizontally at half the Y rate and as a result there are half as many CB and CR values as Y values.

The out of the box show function will not work on this dataset as it does not have Rescale Slope listed in the head so we have to create one

def show_one(file):
    """ function to view a dicom image when Rescale Slope is not noted"""
    pat = dcmread(file)
    trans = Transform(Resize(128))
    dicom_create = PILDicom.create(file)
    dicom_transform = trans(dicom_create)
    return show_image(dicom_transform)

show_one(mitems[0])

<matplotlib.axes._subplots.AxesSubplot at 0x1b0459ffd48>

But why does the image look unnatural?

This is because of the YBR_FULL_422 interpretation so we have to covert the interpretation from YBR_FULL_422 to RGB so that it can look a bit more realistic.

Pydicom provides a means of converting from one color space to another by using convert_color_space where it takes the (pixel array, current color space, desired color space) as attributes. This is done by accessing the pixel_array and then converting to the desired color space

from pydicom.pixel_data_handlers.util import convert_color_space

arr = patient_two.pixel_array
convert = convert_color_space(arr, 'YBR_FULL_422', 'RGB')
show_image(convert)

<matplotlib.axes._subplots.AxesSubplot at 0x1b04b8bc808>

That looks a lot better!

Note: Functions listed with a ✔ are custom functions

DICOM Datasets

Here is a list of 3 DICOM datasets that you can play around with. Each of these 3 datasets have different attributes and shows how there can be a vast difference in what is contained in different DICOM datasets.

the SIIM_SMALL dataset ((250 DICOM files, ~30MB) is conveniently provided in the fastai library but is limited in some of its attributes for example it does not have RescaleIntercept or RescaleSlope and its pixel range is limited in the range of 0 and 255

Kaggle has an easily accessible (437MB) CT medical image dataset from the cancer imaging archive. The dataset consists of 100 images (512px by 512px) with pixel ranges from -2000 to +2000

The Thyroid Segmentation in Ultrasonography Dataset provides low quality (ranging from 253px by 253px) DICOM images where each DICOM image has multiple frames (average of 1000)

#Load the dependancies
from fastai.basics import *
from fastai.callback.all import *
from fastai.vision.all import *
from fastai.medical.imaging import *

import pydicom
import seaborn as sns
matplotlib.rcParams['image.cmap'] = 'bone'
from matplotlib.colors import ListedColormap, LinearSegmentedColormap

Loading DICOMs which have 1 frame per file

The SIIM_SMALL dataset is a DICOM dataset where each DICOM file has a pixel_array that contains 1 image. In this case the show function within fastai.medical.imaging conveniently displays the image

source = untar_data(URLs.SIIM_SMALL)
items = get_dicom_files(source)
patient1 = dcmread(items[0])
patient1.show()

1 frame per file

Loading an image from the CT medical image dataset which also contains 1 frame per DICOM file. This image is a slice of a CT scan looking at the lungs with the heart in the middle.

csource = Path('C:/PillView/NIH/data/dicoms')
citems = get_dicom_files(csource)
patient2 = dcmread(citems[0])
patient2.show()

1 frame per file

However what if a DICOM dataset has multiple frames per DICOM file

Loading DICOMs which have multiple frames per file ✔

The Thyroid Segmentation in Ultrasonography Dataset is a dataset where each DICOMfile has multiple frames per file. Using the same format as above to view an image:

tsource = Path('D:/Datasets/thyroid')
titems = get_dicom_files(tsource)
patient3 = dcmread(titems[0])
#patient3.show()

This will result in a TypeError because the current show function does not have a means of displaying files with multiple frames

Customizing the show function now checks to see if the file contains more than 1 frame and then displays the image accordingly. You can also choose how many frames to view (the default is 1). It was also noted that the show_images function does not accept colormaps and hence that function also had to be slightly modified

#updating to handle colormaps
@delegates(subplots)
def show_images(ims, nrows=1, ncols=None, titles=None, cmap=None, **kwargs):
    "Show all images `ims` as subplots with `rows` using `titles`"
    if ncols is None: ncols = int(math.ceil(len(ims)/nrows))
    if titles is None: titles = [None]*len(ims)
    axs = subplots(nrows, ncols, **kwargs)[1].flat
    for im,t,ax in zip(ims, titles, axs): show_image(im, ax=ax, title=t, cmap=cmap)

#updating to handle multiple frames
@patch
@delegates(show_image, show_images)
def show(self:DcmDataset, frames=1, scale=True, cmap=plt.cm.bone, min_px=-1100, max_px=None, **kwargs):
    px = (self.windowed(*scale) if isinstance(scale,tuple)
          else self.hist_scaled(min_px=min_px,max_px=max_px,brks=scale) if isinstance(scale,(ndarray,Tensor))
          else self.hist_scaled(min_px=min_px,max_px=max_px) if scale
          else self.scaled_px)
    if px.ndim > 2: 
        gh=[]
        p = px.shape; print(f'{p[0]} frames per file')
        for i in range(frames): u = px[i]; gh.append(u)
        show_images(gh, cmap=cmap, **kwargs)    
    else: 
        print('1 frame per file')
        show_image(px, cmap=cmap, **kwargs)

patient3.show(10)

932 frames per file

The images now display the number of frames specified as well as how many frames there are in each file. It also now allows a cmap to be passed in.

patient3.show(10, cmap=plt.cm.ocean)

932 frames per file

This function also works when each DICOM file only has 1 frame

patient2.show()

1 frame per file

Saving files from multiple frames

The Thyroid segmentation dataset is broken down into 2 folders each containing 16 .dcm files each. It would be good to know what the total number of frames are within the dataset.

For this we use a custom function to get the total number of frames in the dataset and how many frames there are in each file

def get_num_frames(source):
    """Get the number of frames in each DICOM"""
    """Some DICOMs have multiple frames and this function helps to find the total number of frames in a DICOM dataset """
    frame_list = []
    h = get_dicom_files(source)
    for i, path in enumerate(h):
        test_im = h[i]
        j = dcmread(test_im) 
        try: 
            v = int(j.NumberOfFrames)
        except: 
            v=1
        frame_list.append(v)
        sl = sum(frame_list); ll = L(frame_list)
    return sl, ll

get_num_frames(tsource)

(31304, (#33) [932,942,1058,1120,958,1064,1134,1060,928,892...])

In this case there are a total of 31304 frames within the dataset with each file having between 800 to 1100 frames. To view a range of frames:

gh = []
for i in range(0,100):
    u = patient3.pixel_array[i,:,:]
    gh.append(u)
show_images(gh, nrows=10, ncols=10)

To save the frames into .jpg format you can use this function to extract frames from 1 file specifying the input source and the output

def get_frames(source, save_path):
    """extract frames from DICOM file and save into specified location"""
    h = get_dicom_files(source)
    test_im = h[0]
    j = dcmread(test_im)
    try:
        frame = int(j.NumberOfFrames)
        print(f'saving files in {save_path}')
    except:
        frame = 0
        print('file has no frames')
    for i in range(frame):
        u = j.pixel_array[i,:,:]
        im = Image.fromarray(u)
        im.save(f'{save_path}/image_{i}.jpg')
    return f'Number of frames saved: {frame}'

get_frames(tsource, tsource/'Test')

saving files in C:\PillView\NIH\data\thyroid\Test

'Number of frames saved: 932'

Another fastai function from_dicoms conveniently converts all the DICOM head attributes into a useable dataframe

t_df = pd.DataFrame.from_dicoms(titems)

Once converted you can now easily view various attributes about the dataset

#Plot 3 comparisons
def plot_comparison(df, feature, feature1, feature2):
    "Plot 3 comparisons from a dataframe"
    fig, (ax1, ax2, ax3) = plt.subplots(1,3, figsize = (16, 4))
    s1 = sns.countplot(df[feature], ax=ax1)
    s1.set_title(feature)
    s2 = sns.countplot(df[feature1], ax=ax2)
    s2.set_title(feature1)
    s3 = sns.countplot(df[feature2], ax=ax3)
    s3.set_title(feature2)
    plt.show()

plot_comparison(t_df, 'PatientSex', 'Columns', 'Rows')

Viewing DICOM `tag` values

To view the tag values within the header you can use the dir() function

patient3.dir()

['AccessionNumber',
 'AcquisitionContextSequence',
 'AcquisitionDateTime',
 'AcquisitionDuration',
 'AcquisitionTimeSynchronized',
 'AnatomicRegionSequence',
 'ApexPosition',
 'BitsAllocated',
 'BitsStored',
 'BoneThermalIndex',
 'BurnedInAnnotation',
 'Columns',
 'ContentDate',
 'ContentTime',
 'CranialThermalIndex',
 'DepthOfScanField',
 'DepthsOfFocus',
 'DeviceSerialNumber',
 'DimensionIndexSequence',
 'DimensionOrganizationSequence',
 'DimensionOrganizationType',
 'FrameOfReferenceUID',
 'HighBit',
 'ImageComments',
 'ImageOrientationPatient',
 'ImagePositionPatient',
 'ImageType',
 'InstanceNumber',
 'InstitutionName',
 'InstitutionalDepartmentName',
 'LargestImagePixelValue',
 'LossyImageCompression',
 'Manufacturer',
 'ManufacturerModelName',
 'MechanicalIndex',
 'Modality',
 'NumberOfFrames',
 'OperatorsName',
 'PatientBirthDate',
 'PatientBirthTime',
 'PatientID',
 'PatientName',
 'PatientOrientation',
 'PatientSex',
 'PerFrameFunctionalGroupsSequence',
 'PhotometricInterpretation',
 'PixelData',
 'PixelRepresentation',
 'PixelSpacing',
 'PositionMeasuringDeviceUsed',
 'PositionReferenceIndicator',
 'PresentationLUTShape',
 'ReferringPhysicianName',
 'RescaleIntercept',
 'RescaleSlope',
 'Rows',
 'SOPClassUID',
 'SOPInstanceUID',
 'SamplesPerPixel',
 'SeriesDate',
 'SeriesDescription',
 'SeriesInstanceUID',
 'SeriesNumber',
 'SeriesTime',
 'SharedFunctionalGroupsSequence',
 'SmallestImagePixelValue',
 'SoftTissueThermalIndex',
 'SoftwareVersions',
 'SpacingBetweenSlices',
 'StationName',
 'StudyDate',
 'StudyID',
 'StudyInstanceUID',
 'StudyTime',
 'SynchronizationFrameOfReferenceUID',
 'SynchronizationTrigger',
 'TransducerApplicationCodeSequence',
 'TransducerBeamSteeringCodeSequence',
 'TransducerGeometryCodeSequence',
 'TransducerScanPatternCodeSequence',
 'TransducerType',
 'UltrasoundAcquisitionGeometry',
 'ViewCodeSequence',
 'VolumeFrameOfReferenceUID',
 'VolumeToTransducerMappingMatrix']

Its worth pointing out that there may be additional information (however not always the case) where there may be a tag for ImageComments. This tag may contain information that may be useful in the modelling. In this case not really useful information except that MeVisLab was used to probably generate the image

patient3.ImageComments

'MeVisLab'

Understanding Tissue Densities

There are a couple of downsides to the SIMM_SLIM dataset because it does not have a number of attributes including RescaleIntercept and RescaleSlope and its pixel distribution is limited to between 0 and 255 pixels. In reality DICOM images have a far more wide-spread range of pixel values.

By using the CT medical image dataset we can now play around with other useful fastai.medical.imaging functionality.

patient2 is an image from this dataset

tensor_dicom = pixels(patient2) #convert into tensor

print(f'RescaleIntercept: {patient2.RescaleIntercept:1f}\nRescaleSlope: {patient2.RescaleSlope:1f}\nMax pixel: '
      f'{tensor_dicom.max()}\nMin pixel: {tensor_dicom.min()}\nShape: {tensor_dicom.shape}')

RescaleIntercept: -1024.000000
RescaleSlope: 1.000000
Max pixel: 1918.0
Min pixel: 0.0
Shape: torch.Size([512, 512])

In this image the RescaleIntercept is -1024, the RescaleSlope is 1, the max and min pixels are 1918 and 0 respectively and the image size is 512 by 512

Plotting a histogram of pixel intensities you can see where the bulk of pixels are located

plt.hist(tensor_dicom.flatten(), color='c')

(array([1.73881e+05, 1.82440e+04, 4.70700e+03, 3.24000e+03, 1.53940e+04,
        3.50250e+04, 7.98600e+03, 2.62500e+03, 8.99000e+02, 1.43000e+02]),
 array([   0. ,  191.8,  383.6,  575.4,  767.2,  959. , 1150.8, 1342.6,
        1534.4, 1726.2, 1918. ], dtype=float32),
 <a list of 10 Patch objects>)

The histogram shows that the minimal pixel value is 0 and the maximum pixel value is 1918. The histogram is predominantly bi-modal with the majority of pixels between the 0 and 100 pixels and between 750 and 1100 pixels.

This image has a RescaleIntercept of -1024 and a RescaleSlope of 1. These two values allows for transforming pixel values into Hounsfield Units(HU). Densities of different tissues on CT scans are measured in HUs

Most CT scans range from -1000HUs to +1000HUs where water is 0HUs, air is -1000HUs and the denser the tissues the higher the HU value. Metals have a much higher HU range +2000HUs so for medical imaging a range of -1000 to +1000HUs is suitable

The pixel values above do not correctly correspond to tissue densities. For example most of the pixels are between pixel values 0 and 100 which correspond to water but this image is predominantly showing the lungs which are filled with air. Air on the Hounsfield scale is -1000 HUs.

This is where RescaleIntercept and RescaleSlope are important. Fastai provides a convenient way scaled_px to rescale the pixels with respect to RescaleIntercept and RescaleSlope.

rescaled pixel = pixel * RescaleSlope + RescaleIntercept

tensor_dicom_scaled = scaled_px(patient2) #convert into tensor taking RescaleIntercept and RescaleSlope into consideration
plt.hist(tensor_dicom_scaled.flatten(), color='c')

(array([1.73881e+05, 1.82440e+04, 4.70700e+03, 3.24000e+03, 1.53940e+04,
        3.50250e+04, 7.98600e+03, 2.62500e+03, 8.99000e+02, 1.43000e+02]),
 array([-1024. ,  -832.2,  -640.4,  -448.6,  -256.8,   -65. ,   126.8,
          318.6,   510.4,   702.2,   894. ], dtype=float32),
 <a list of 10 Patch objects>)

print(f'Max pixel: {tensor_dicom_scaled.max()}\nMin pixel: {tensor_dicom_scaled.min()}')

Max pixel: 894.0
Min pixel: -1024.0

After re-scaling the maximum pixel value is 894 and the minimum value is -1024 and we can now correctly see what parts of the image correspond to what parts of the body based on the Hounsfield scale.

Looking at the top end of the histogram what does the image look like with values over 300 HUs?

The show function has the capability of specifying max and min values

patient2.show(max_px=894, min_px=300, figsize=(5,5))

1 frame per file

HU values above +300 typically will show the bone stuctures within the image

patient2.show(max_px=250, min_px=-250, figsize=(5,5))

1 frame per file

Within this range you can now see the aorta and the parts of the heart(image middle) as well as muscle and fat.

patient2.show(max_px=-250, min_px=-600, figsize=(5,5))

1 frame per file

In this range you just make out outlines. The histogram does show that within this range there are not many pixels

patient2.show(max_px=-600, min_px=-1000, figsize=(5,5))

1 frame per file

Within this range you can clearly see the bronchi within the lungs

patient2.show(max_px=-900, min_px=-1024, figsize=(5,5))

1 frame per file

At this range you can now also clearly see the curve of the scanner.

The show function by default has a max_px value of None and a min_px value of -1100

patient2.show(max_px=None, min_px=-1100, figsize=(5,5))

1 frame per file

Image re-scaling as done above is really for the benefit of humans. Computer screens can display about 256 shades of grey and the human eye is only capable of detecting about a 6% change in greyscale (ref) meaning the human eye can only detect about 17 different shades of grey.

DICOM images may have a wide range from -1000 to +1000 and for humans to be able to see relevant structures within the image a process of windowing is utilized. Fastai provides various dicom_windows so that only specific HU values are displayed on the screen. More about windowing can be found here

Bins

Where windowing is for the benefit of the human, computers produce better results from training when the data has a uniform distribution as mentioned in an article by Jeremy don't see like a radiologist

Looking back at the pixel distribution we can see that the image does not have a uniform distribution

plt.hist(tensor_dicom_scaled.flatten(), color='c')

(array([1.73881e+05, 1.82440e+04, 4.70700e+03, 3.24000e+03, 1.53940e+04,
        3.50250e+04, 7.98600e+03, 2.62500e+03, 8.99000e+02, 1.43000e+02]),
 array([-1024. ,  -832.2,  -640.4,  -448.6,  -256.8,   -65. ,   126.8,
          318.6,   510.4,   702.2,   894. ], dtype=float32),
 <a list of 10 Patch objects>)

fastai has a function freqhist_hist that splits the range of pixel values into groups depending on what value you set for n_bins, such that each group has around the same number of pixels.

For example if you set n_bins to 1, the pixel values are split into 2 distinct pixel bins

ten_freq = tensor_dicom_scaled.freqhist_bins(n_bins=1)
fh = patient2.hist_scaled(ten_freq)
plt.hist(ten_freq.flatten(), color='c'); show_image(fh, figsize=(7,7))

<matplotlib.axes._subplots.AxesSubplot at 0x23dc63fbf48>

In this case you see the 2 polar sides of the image at -1000HUs you see the air portions and at 500HUs you see the bone structures clearly but the distribution is still not fully acceptable for the machine learning model.

with n_bins at 100(this is the default number used by show)

ten_freq2 = tensor_dicom_scaled.freqhist_bins(n_bins=100)
fh2 = patient2.hist_scaled(ten_freq2)
plt.hist(ten_freq2.flatten(), color='c'); show_image(fh2, figsize=(7,7))

<matplotlib.axes._subplots.AxesSubplot at 0x23dc4bb62c8>

with n_bins at 100000 the pixels are showing a more uniform distribution

ten_freq3 = tensor_dicom_scaled.freqhist_bins(n_bins=100000)
fh3 = patient2.hist_scaled(ten_freq3)
plt.hist(ten_freq3.flatten(), color='c'); show_image(fh3, figsize=(7,7))

<matplotlib.axes._subplots.AxesSubplot at 0x23db3a86848>

What effect does this have on training outcomes. That will be the topic of the next blog

fin

Note: Functions listed with a ✔ are custom functions

Goal:

The goals of this starter notebook are to:

use DICOMs as the image input
high level overview of what considerations need to be taken and what the results mean when creating a model that predicts medical conditions

The dataset used is conveniently provided by fastai - SIIM-ACR Pneumothorax Segmentation dataset and contains 250 Dicom images (175 No Pneumothorax and 75 Pneumothorax)

Considerations:

patient overlap
sampling
evaluting AI models for medical use

#load dependancies
from fastai.basics import *
from fastai.callback.all import *
from fastai.vision.all import *
from fastai.medical.imaging import *

import pydicom
import seaborn as sns
matplotlib.rcParams['image.cmap'] = 'bone'

Load the Data

pneumothorax_source = untar_data(URLs.SIIM_SMALL)
items = get_dicom_files(pneumothorax_source, recurse=True, folders='sm')
df = pd.read_csv(pneumothorax_source/f"labels_sm.csv")

items

(#26) [Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000000 - Copy.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000000.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000002.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000005.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000006 - Copy.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000006.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000007.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000008.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000009.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/sm/No Pneumothorax/000011.dcm')...]

Note: The SIIM_SMALL dataset has no duplicate patient IDs, has an equal number of males and females so I am using a custom even smaller dataset to show the functionality of DicomSplit and DataSplit below

Viewing the Data

View Dicom

The show function is specifically tailored to display .dcm formats. By customizing the show function we have now view patient information with each image

@patch
@delegates(show_image)
def show_dinfo(self:DcmDataset, scale=True, cmap=plt.cm.bone, min_px=-1100, max_px=None, **kwargs):
    """show function that prints patient attributes from DICOM head"""
    px = (self.windowed(*scale) if isinstance(scale,tuple)
          else self.hist_scaled(min_px=min_px,max_px=max_px,brks=scale) if isinstance(scale,(ndarray,Tensor))
          else self.hist_scaled(min_px=min_px,max_px=max_px) if scale
          else self.scaled_px)
    print(f'Patient Age: {self.PatientAge}')
    print(f'Patient Sex: {self.PatientSex}')
    print(f'Body Part Examined: {self.BodyPartExamined}')
    print(f'Rows: {self.Rows} Columns: {self.Columns}')
    show_image(px, cmap=cmap, **kwargs)

patient = 7
sample = dcmread(items[patient])
sample.show_dinfo()

Patient Age: 31
Patient Sex: M
Body Part Examined: CHEST
Rows: 1024 Columns: 1024

Create a Dataframe of all the `tags` in the header section

Tip: The head section of a DICOM images contains alot of useful information (known as tags) and fastai provides a conveninent way by using the from_dicoms function of getting that information and placing it into a DataFrame. However we do not need to have all the information ported to a Dataframe

For example we can create a DataFrame of all the tags in the header section. In this case there are 42 columns and in most cases we do not need all this information

full_dataframe = pd.DataFrame.from_dicoms(items)
full_dataframe[:1]

We can create a custom dataframe that takes into consideration the information we want, for example:

filename
age
sex
row size
column size

#updating to accomodate 
def _dcm2dict2(fn, **kwargs): 
        t = fn.dcmread()
        return fn, t.PatientID, t.PatientAge, t.PatientSex, t.Rows, t.Columns

@delegates(parallel)
def _from_dicoms2(cls, fns, n_workers=0, **kwargs):
    return pd.DataFrame(parallel(_dcm2dict2, fns, n_workers=n_workers, **kwargs))
pd.DataFrame.from_dicoms2 = classmethod(_from_dicoms2)

test_df = pd.DataFrame.from_dicoms2(items)
test_df.columns=['file', 'PatientID', 'Age', 'Sex', 'Rows', 'Cols']
test_df.to_csv('test_df.csv')
test_df.head()

EDA

We can do some exploratory data analysis and see that this custom dataset has duplicate patient IDs

#Plot 2 comparisons
def plot_comparison(df, feature, feature1):
    "Plot 3 comparisons from a dataframe"
    fig, (ax1, ax2) = plt.subplots(1,2, figsize = (16, 4))
    s1 = sns.countplot(df[feature], ax=ax1)
    s1.set_title(feature)
    s2 = sns.countplot(df[feature1], ax=ax2)
    s2.set_title(feature1)
    plt.show()

plot_comparison(test_df, 'PatientID', 'Sex')

Age comparison

def age_comparison(df, feature):
    "Plot hisogram of age range in dataset"
    fig, (ax1) = plt.subplots(1,1, figsize = (16, 4))
    s1 = sns.countplot(df[feature], ax=ax1)
    s1.set_title(feature)
    plt.show()
    
age_comparison(test_df, 'Age')

Modelling

Some considerations when modelling the data:

is there patient overlap between train and validation sets
sampling - how many negative and postive cases are represented in the train/val split
augmentations - consideration of what augmentations are used and why in some cases may not be useful

Patient Overlap

It is important to know if there is going to be any patient overlap when creating the train and validation sets as this may lead to an overly optimistic result when evaluating against a test set. The great thing about DICOMs is that we can check to see if there are any duplicate patientIDs in the test and valid sets when we split our data

DicomSplit ✔

DicomSplit is a custom function that uses the default fastai splitting function that splits the data into train and validation sets based on valid_pct value but now also checks to see if identical patient IDs exist in both the train and validation sets.

def DicomSplit(valid_pct=0.2, seed=None, **kwargs):
    "Splits `items` between train/val with `valid_pct`"
    "and checks if identical patient IDs exist in both the train and valid sets"
    def _inner(o, **kwargs):
        train_list=[]; valid_list=[]
        if seed is not None: torch.manual_seed(seed)
        rand_idx = L(int(i) for i in torch.randperm(len(o)))
        cut = int(valid_pct * len(o))
        trn = rand_idx[cut:]; trn_p = o[rand_idx[cut:]]
        val = rand_idx[:cut]; val_p = o[rand_idx[:cut]]
        
        for i, im in enumerate(trn_p):
            trn = im.dcmread()
            patient_ID = trn.PatientID
            train_list.append(patient_ID)
        for j, jm in enumerate(val_p):
            val = jm.dcmread()
            vpatient_ID = val.PatientID
            valid_list.append(vpatient_ID) 
        print(set(train_list) & set(valid_list))
        return rand_idx[cut:], rand_idx[:cut]
    return _inner

Tip: There are a number of ways of setting the seed to ensure reproducible results. The easiest way in fastai is to use set_seed or you could incorporate it within a function

set_seed(7)

trn,val = DicomSplit(valid_pct=0.2)(items)
trn, val

{'6224213b-a185-4821-8490-c9cba260a959'}

((#21) [2,13,9,12,11,24,8,14,16,6...], (#5) [19,18,3,23,17])

The custom test dataset only has 26 images (small number of images to show how DicomSplit works) which is split into a test set of 21 and a valid set of 5 using valid_pct of 0.2. By customizing RandomSplitter in DicomSplit you can check to see if there are any duplicate PatientIDs betweeen the 2 sets.

In this case there is a duplicate ID: 6224213b-a185-4821-8490-c9cba260a959, this patient is present in both the train and validation sets.

Important: When working with a medical data set, is it important to consider that the splits should be based on patient identifiers, and not on the individual examples.

Sampling

This dataset consists of 2 classes Pneumothorax and No Pneumothorax. It is important to consider how this data is represented within the train and validation sets.

DataSplit ✔

DataSplit looks at how many Pneumothorax and No Pneumothorax images there are in the training and validation sets. This is to view how fair the train/val split is to ensure good model sampling.

def DataSplit(valid_pct=0.2, seed=None, **kwargs):
    "Check the number of each class in train and valid sets"
    def _inner(o, **kwargs):
        train_list=[]; valid_list=[]
        if seed is not None: torch.manual_seed(seed)
        rand_idx = L(int(i) for i in torch.randperm(len(o)))
        cut = int(valid_pct * len(o)) 
        trn_p = o[rand_idx[cut:]]
        val_p = o[rand_idx[:cut]]
        
        for p in enumerate(trn_p):
            b = str(p).split('/')[7]
            train_list.append(b)
        for q in enumerate(val_p):
            e = str(q).split('/')[7]
            valid_list.append(e)
        train_totals = {x:train_list.count(x) for x in train_list}
        valid_totals = {x:valid_list.count(x) for x in valid_list}
        print(f'Train totals: {train_totals}\nValid totals: {valid_totals}')
        return rand_idx[cut:], rand_idx[:cut]
    return _inner

We can now see the how the data is split using set_seed. For example if set it to 7 like the prior example:

set_seed(7)

trn,val = DataSplit(valid_pct=0.2)(items)
trn, val

Train totals: {'No Pneumothorax': 16, 'Pneumothorax': 5}
Valid totals: {'Pneumothorax': 3, 'No Pneumothorax': 2}

((#21) [2,13,9,12,11,24,8,14,16,6...], (#5) [19,18,3,23,17])

In this case the train set has 16 No Pneumothorax and 5 Pneumothorax images and the valid set has 2 No Pneumothorax and 3 Pneumothorax images

How about using a seed of 77

set_seed(77)

trn,val = DataSplit(valid_pct=0.2)(items)
trn, val

Train totals: {'No Pneumothorax': 14, 'Pneumothorax': 7}
Valid totals: {'Pneumothorax': 1, 'No Pneumothorax': 4}

((#21) [9,18,8,10,12,17,3,1,20,22...], (#5) [19,6,14,7,5])

In this case the train set has 14 No Pneumothorax and 7 Pneumothorax images and the valid set has 4 No Pneumothorax and 1 Pneumothorax image

Note: In these cases we can see what differences setting the seed has on the distribution of images within the training and validation sets

Note: >>Work in progress work on various techniques to help with unbalanced datasets especially true for medical image datasets where there are typically alot more images of ’normal’ compared to ’diseased’ (oversampling, undersampling, stratified k-fold cross validation etc)

Augmentations

img1 = (pneumothorax_source/'chest1.png'); img2 = (pneumothorax_source/'chest2.png')

Note: >>Work in progress Choosing the right augmentations is important in determing how it affects the sampling process. For example in some cases it may not be a good idea to flip images.

Here is an image of a 'normal' patient in its correct orientation (heart showing in the middle right)

Image.open(img1)

If we flip the image

Image.open(img2)

We can now see the heart is middle left. If the classifier was looking to detect defects of the heart then this type of augmentation would not be suitable.

Training

Tip: Garbage Collector is a great way of freeing up memory if needed

#clear out some memory
import gc
gc.collect()

12900

Switching back to the full dataset which contains 250 images.

items_full = get_dicom_files(pneumothorax_source, recurse=True, folders='train')
df_full = pd.read_csv(pneumothorax_source/f"labels.csv")

Using DataSplit we can check the number of each class represented in the training and validation sets

set_seed(7)

trn,val = DataSplit(valid_pct=0.2)(items_full)
trn, val

Train totals: {'No Pneumothorax': 141, 'Pneumothorax': 59}
Valid totals: {'No Pneumothorax': 34, 'Pneumothorax': 16}

((#200) [33,65,231,167,74,127,184,89,122,79...],
 (#50) [115,233,139,163,161,177,57,21,34,99...])

Create the DataBlock and specify some transforms

xtra_tfms = [RandomResizedCrop(194)]
batch_tfms = [*aug_transforms(do_flip=False, flip_vert=False, xtra_tfms=xtra_tfms), Normalize.from_stats(*imagenet_stats)]

For the Splitter we have already split the data into training and validation sets using DicomSplit and we now need to incorporate this split into the DataBlock. Fastai provides a convenient method using IndexSplitter which splits items so that val_idx are in the validation set and the others in the training set. DicomSplit splits the indexs of the dataset so we can feed this into IndexSplitter with val_idx set to the val index that was created with DicomSplit

pneumothorax = DataBlock(blocks=(ImageBlock(cls=PILDicom), CategoryBlock),
                   get_x=lambda x:f'{pneumothorax_source}/{x[0]}',
                   get_y=lambda x:x[1],
                   splitter=IndexSplitter(val),
                   item_tfms=Resize(256),
                   batch_tfms=batch_tfms)

dls = pneumothorax.dataloaders(df_full.values, bs=16, num_workers=0)

Check train and valid sizes

len(dls.train_ds), len(dls.valid_ds)

(200, 50)

#net = xresnext50(pretrained=False, sa=True, act_cls=Mish, n_out=dls.c)
net = resnet50()
learn = Learner(dls, 
                net,
                metrics=[accuracy], 
                cbs=[ShowGraphCallback()])

Tip: If you do not specifiy a loss function or optimization function fastai automatically allocates one. You can view the loss_func and opt_func as follows:

learn.loss_func

FlattenedLoss of CrossEntropyLoss()

learn.opt_func

<function fastai.optimizer.Adam(params, lr, mom=0.9, sqr_mom=0.99, eps=1e-05, wd=0.01, decouple_wd=True)>

learn.lr_find()

SuggestedLRs(lr_min=0.012022644281387329, lr_steep=0.0003311311302240938)

learn.unfreeze()
learn.fit_one_cycle(3, slice(1e-2))

show_results

To see how show_results works, I slightly tweaked it so that we can see the ground truth, predictions and the probabilites.

Tip: get_preds returns a tuple: Probabilites, Ground Truth, Prediction

def show_results2(self, ds_idx=1, dl=None, max_n=9, shuffle=False, **kwargs):
    if dl is None: dl = self.dls[ds_idx].new(shuffle=shuffle)
    b = dl.one_batch()
    p,g,preds = self.get_preds(dl=[b], with_decoded=True)
    print(f'Ground Truth: {g}\n Probabilites: {p}\n Prediction: {preds}\n')
    self.dls.show_results(b, preds, max_n=max_n, **kwargs)

show_results2(learn, max_n=12, nrows=2, ncols=6)

Ground Truth: TensorCategory([0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0], dtype=torch.int32)
 Probabilites: tensor([[6.5734e-01, 3.4266e-01, 2.0420e-10,  ..., 2.2660e-10, 2.3010e-10,
         2.3564e-10],
        [5.9493e-01, 4.0506e-01, 1.7759e-08,  ..., 1.7563e-08, 1.9316e-08,
         1.9565e-08],
        [8.7132e-01, 1.2868e-01, 3.4132e-11,  ..., 3.7495e-11, 3.8186e-11,
         4.0348e-11],
        ...,
        [7.7597e-01, 2.2403e-01, 1.9229e-09,  ..., 2.0393e-09, 1.9173e-09,
         2.0422e-09],
        [6.5044e-01, 3.4956e-01, 5.0422e-12,  ..., 6.0412e-12, 5.9057e-12,
         6.2554e-12],
        [7.3728e-01, 2.6272e-01, 2.0523e-14,  ..., 2.5750e-14, 2.8775e-14,
         2.8507e-14]])
 Prediction: tensor([0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0])

The probabilies determine the predicted outcome. In this dataset there are only 2 classes No Pneumothorax and Pneumothorax hence the reason why each probability has 2 values, the first value is the probability whether the image belongs to class 0 or No Pneumothorax and the second value is the probability whether the image belongs to class 1 or Pneumothorax

Tip: It is a good idea looking at the probabilities to see how certain they are. For example: [0.4903, 0.5097] and [0.1903, 0.8097] both produce the same results that the image belongs to class 1 but in the second case the model is alot more certain that it belongs to class 1.

Model Evaluation

Because medical models are high impact it is important to know how good a model is at detecting a certain condition.

Accuracy

What is the accuracy of the model above? You can simply use accuracy to get that information

accuracy

0.6651984126984127

The above model has an accuracy of 66.5%.

Accuracy is the probablity that the model is correct or to be more specific:

Accuracy is the probability that the model is correct and the patient has the condition PLUS the probability that the model is correct and the patient does not have the condition

False Positive & False Negative

There are some other key terms that need to be used when evaluating medical models:

False Negative is an error in which a test result improperly indicates no presence of a condition (the result is negative), when in reality it is present.

False Positive is an error in which a test result improperly indicates presence of a condition, such as a disease (the result is positive), when in reality it is not present

Sensitivity & Specificity

Sensitivity or True Positive Rate is where the model classifies a patient has the disease given the patient actually does have the disease. Sensitivity quantifies the avoidance of false negatives

Example: A new test was tested on 10,000 patients, if the new test has a sensitivity of 90% the test will correctly detect 9,000 (True Positive) patients but will miss 1000 (False Negative) patients that have the condition but were tested as not having the condition

Specificity or True Negative Rate is where the model classifies a patient as not having the disease given the patient actually does not have the disease. Specificity quantifies the avoidance of false positives

Tip: Understanding and using sensitivity, specificity and predictive values is a great paper if you are interested in learning more

PPV and NPV

Most medical testing is evaluated via PPV (Postive Predictive Value) or NPV (Negative Predictive Value).

PPV - if the model predicts a patient has a condition what is probabilty that the patient actually has the condition
NPV - if the model predicts a patient does not have a condition what is the probability that the patient actually does not have the condition

The ideal value of the PPV, with a perfect test, is 1 (100%), and the worst possible value would be zero

The ideal value of the NPV, with a perfect test, is 1 (100%), and the worst possible value would be zero

Confusion Matrix

Plot a confusion matrix - note that this is plotted against the valid dataset

interp = ClassificationInterpretation.from_learner(learn)
losses,idxs = interp.top_losses()
len(dls.valid_ds)==len(losses)==len(idxs)
interp.plot_confusion_matrix(figsize=(7,7))

We can manually reproduce the results interpreted from plot_confusion_matrix

upp, low = interp.confusion_matrix()
tn, fp = upp[0], upp[1]
fn, tp = low[0], low[1]
print(tn, fp, fn, tp)

29 7 10 4

Note: Sensitivity = True Positive/(True Positive + False Negative)

sensitivity = tp/(tp + fn)
sensitivity

0.2857142857142857

In this case the model only has a sensitivity of 28% and hence is only capable of correctly detecting 28% True Positives(ie who have Pneumothorax) but will miss 72% of False Negatives (patients that actually have Pneumothorax but were told they did not! Not a good situation to be in).

Note: This is also know as a Type II error

Note: Specificity = True Negative/(False Positive + True Negative)

specificity = tn/(fp + tn)
specificity

0.8055555555555556

In this case the model has a specificity of 80% and hence can correctly detect 80% of the time that a patient does NOT have Pneumothorax but will incorrectly classify that 20% of the patients have Pneumothorax (False Postive) but actually do not.

Note: This is also known as a Type I error

Positive Predictive Value (PPV)

ppv = tp/(tp+fp)
ppv

0.36363636363636365

In this case the model performs poorly in correctly predicting patients with Pneumothorax

Negative Predictive Value (NPV)

npv = tn/(tn+fn)
npv

0.7435897435897436

This model is better at predicting patients with No Pneumothorax

Some of these metrics can be calculated using sklearn's classification report

interp.print_classification_report()

                 precision    recall  f1-score   support

No Pneumothorax       0.74      0.81      0.77        36
   Pneumothorax       0.36      0.29      0.32        14

       accuracy                           0.66        50
      macro avg       0.55      0.55      0.55        50
   weighted avg       0.64      0.66      0.65        50

Calculating Accuracy

The accuracy of this model as mentioned before is 66% - lets now calculate this!

We can also look at Accuracy as:

Tip: accuracy = sensitivity x prevalence + specificity * (1 - prevalence)

Prevalence is a statistical concept referring to the number of cases of a disease that are present in a particular population at a given time.

The prevalence in this case is how many patients in the valid dataset have the condition compared to the total number. To view the number of Pneuomothorax patients in the valid set

t= dls.valid_ds.cat
#t[0]

There are 20 Pneumothorax images in the valid set hence the prevalance here is 20/75 = 0.27

accuracy = (sensitivity * 0.27) + (specificity * (1 - 0.27))
accuracy

0.6651984126984127

fin

Note: This is based on notebook 60_medical_imaging which can be found here

Import the libraries

from fastai.basics import *
from fastai.vision.all import *
from fastai.data.transforms import *
from fastai.medical.imaging import *

import pydicom,kornia,skimage

try:
    import cv2
    cv2.setNumThreads(0)
except: pass

import seaborn as sns
sns.set(style="whitegrid")
sns.set_context("paper")

#Load the Data
pneumothorax_source = untar_data(URLs.SIIM_SMALL)

Patching

get_dicom_files

Provides a convenient way of recursively loading .dcm images from a folder. By default the folders option is set to False but you could specify a specific folder if required

#get dicom files
items = get_dicom_files(pneumothorax_source, recurse=True, folders='train')
items

(#250) [Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000000.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000002.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000005.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000006.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000007.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000008.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000009.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000011.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000012.dcm'),Path('C:/Users/avird/.fastai/data/siim_small/train/No Pneumothorax/000014.dcm')...]

dcmread

Pydicom is a python package for parsing DICOM files and makes it easy to covert DICOM files into pythonic structures for easier manipulation. Files are opened using pydicom.dcmread

img = items[10]
dimg = dcmread(img)
type(dimg)

pydicom.dataset.FileDataset

Understanding Dicoms

Tip: This is a quick overview of understanding dicoms

You can now view all the information within the DICOM file. Explanation of each element is beyond the scope of this tutorial but this site has some excellent information about each of the entries. Information is listed by the DICOM tag (eg: 0008, 0005) or DICOM keyword (eg: Specific Character Set)

dimg

(0008, 0005) Specific Character Set              CS: 'ISO_IR 100'
(0008, 0016) SOP Class UID                       UI: Secondary Capture Image Storage
(0008, 0018) SOP Instance UID                    UI: 1.2.276.0.7230010.3.1.4.8323329.6340.1517875197.696624
(0008, 0020) Study Date                          DA: '19010101'
(0008, 0030) Study Time                          TM: '000000.00'
(0008, 0050) Accession Number                    SH: ''
(0008, 0060) Modality                            CS: 'CR'
(0008, 0064) Conversion Type                     CS: 'WSD'
(0008, 0090) Referring Physician's Name          PN: ''
(0008, 103e) Series Description                  LO: 'view: AP'
(0010, 0010) Patient's Name                      PN: '13f40bdc-803d-4fe0-b008-21234c2be1c3'
(0010, 0020) Patient ID                          LO: '13f40bdc-803d-4fe0-b008-21234c2be1c3'
(0010, 0030) Patient's Birth Date                DA: ''
(0010, 0040) Patient's Sex                       CS: 'F'
(0010, 1010) Patient's Age                       AS: '74'
(0018, 0015) Body Part Examined                  CS: 'CHEST'
(0018, 5101) View Position                       CS: 'AP'
(0020, 000d) Study Instance UID                  UI: 1.2.276.0.7230010.3.1.2.8323329.6340.1517875197.696623
(0020, 000e) Series Instance UID                 UI: 1.2.276.0.7230010.3.1.3.8323329.6340.1517875197.696622
(0020, 0010) Study ID                            SH: ''
(0020, 0011) Series Number                       IS: "1"
(0020, 0013) Instance Number                     IS: "1"
(0020, 0020) Patient Orientation                 CS: ''
(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0010) Rows                                US: 1024
(0028, 0011) Columns                             US: 1024
(0028, 0030) Pixel Spacing                       DS: [0.168, 0.168]
(0028, 0100) Bits Allocated                      US: 8
(0028, 0101) Bits Stored                         US: 8
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 2110) Lossy Image Compression             CS: '01'
(0028, 2114) Lossy Image Compression Method      CS: 'ISO_10918_1'
(7fe0, 0010) Pixel Data                          OB: Array of 118256 elements

Some key pointers on the tag information above:

Pixel Data (7fe0 0010) - This is where the raw pixel data is stored. The order of pixels encoded for each image plane is left to right, top to bottom, i.e., the upper left pixel (labeled 1,1) is encoded first
Photometric Interpretation (0028, 0004) - aka color space. In this case it is MONOCHROME2 where pixel data is represented as a single monochrome image plane where low values=dark, high values=bright. If the colorspace was MONOCHROME then the low values=bright and high values=dark info.
Samples per Pixel (0028, 0002) - This should be 1 as this image is monochrome. This value would be 3 if the color space was RGB for example
Bits Stored (0028 0101) - Number of bits stored for each pixel sample
Pixel Represenation (0028 0103) - can either be unsigned(0) or signed(1). The default is unsigned. This Kaggle notebook by Jeremy explains why BitsStored and PixelRepresentation are important
Lossy Image Compression (0028 2110) - 00 image has not been subjected to lossy compression. 01 image has been subjected to lossy compression.
Lossy Image Compression Method (0028 2114) - states the type of lossy compression used (in this case JPEG Lossy Compression)
Pixel Data (7fe0, 0010) - Array of 118256 elements represents the image pixel data that pydicom uses to convert the pixel data into an image.

Important tags not included in this dataset:

Rescale Intercept (0028, 1052) - The value b in relationship between stored values (SV) and the output units. Output units = m*SV + b.
Rescale Slope (0028, 1053) - m in the equation specified by Rescale Intercept (0028,1052).

The Rescale Intercept and Rescale Slope are applied to transform the pixel values of the image into values that are meaningful to the application. These two values allows for transforming pixel values into Hounsfield Units(HU). Densities of different tissues on CT scans are measured in HUs

The Hounsfield scale is a quantitative scale for describing radiodensity in medical CT and provides an accurate density for the type of tissue. On the Hounsfield scale, air is represented by a value of −1000 (black on the grey scale) and bone between +300 (cancellous bone) to +3000 (dense bone) (white on the grey scale), water has a value of 0 HUs and metals have a much higher HU range +2000 HUs.

The pixel values in the histogram above do not correctly correspond to tissue densities. For example most of the pixels are between pixel values 0 and 200 which correspond to water but this image is predominantly showing the lungs which are filled with air. Air on the Hounsfield scale is -1000 HUs.

This is where RescaleIntercept and RescaleSlope are important. Fastai provides a convenient way by using a function scaled_px to rescale the pixels with respect to RescaleIntercept and RescaleSlope.

rescaled pixel = pixel * RescaleSlope + RescaleIntercept

By default pydicom reads pixel data as the raw bytes found in the file and typically PixelData is often not immediately useful as data may be stored in a variety of different ways:

The pixel values may be signed or unsigned integers, or floats
There may be multiple image frames
There may be multiple planes per frame (i.e. RGB) and the order of the pixels may be different These are only a few examples and more information can be found on the pycidom website

What does pixel data look like?

dimg.PixelData[:200]

b'\xfe\xff\x00\xe0\x00\x00\x00\x00\xfe\xff\x00\xe0\xe0\xcd\x01\x00\xff\xd8\xff\xdb\x00C\x00\x03\x02\x02\x02\x02\x02\x03\x02\x02\x02\x03\x03\x03\x03\x04\x06\x04\x04\x04\x04\x04\x08\x06\x06\x05\x06\t\x08\n\n\t\x08\t\t\n\x0c\x0f\x0c\n\x0b\x0e\x0b\t\t\r\x11\r\x0e\x0f\x10\x10\x11\x10\n\x0c\x12\x13\x12\x10\x13\x0f\x10\x10\x10\xff\xc0\x00\x0b\x08\x04\x00\x04\x00\x01\x01\x11\x00\xff\xc4\x00\x1d\x00\x00\x02\x02\x03\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x04\x02\x05\x00\x01\x06\x07\x08\t\xff\xc4\x00]\x10\x00\x01\x04\x01\x03\x02\x05\x01\x04\x05\x05\n\x08\x0b\x05\t\x01\x00\x02\x03\x11\x04\x12!1\x05A\x06\x13"Qaq\x142\x81\x91\x07#B\xa1\xb1\x15R\xc1\xd1\xd2\x08\x16$3b\x92\x95\xb2\xb3\xe1%CSr\x82\x93\xa2'

Because of the complexity in interpreting PixelData, pydicom provides an easy way to get it in a convenient form: pixel_array which returns a numpy.ndarray containing the pixel data:

dimg.pixel_array, dimg.pixel_array.shape

(array([[  2,   6,   5, ...,   3,   3,   2],
        [  5,   9,   8, ...,   6,   5,   5],
        [  5,   9,   9, ...,   6,   5,   5],
        ...,
        [ 49,  85,  80, ..., 123, 121,  69],
        [ 54,  88,  81, ..., 118, 115,  70],
        [ 17,  48,  39, ...,  46,  52,  27]], dtype=uint8), (1024, 1024))

Class TensorDicom

Inherits from TensorImage and converts the pixel_array into a TensorDicom

ten_img = TensorDicom(dimg.pixel_array)
ten_img

TensorDicom([[  2,   6,   5,  ...,   3,   3,   2],
        [  5,   9,   8,  ...,   6,   5,   5],
        [  5,   9,   9,  ...,   6,   5,   5],
        ...,
        [ 49,  85,  80,  ..., 123, 121,  69],
        [ 54,  88,  81,  ..., 118, 115,  70],
        [ 17,  48,  39,  ...,  46,  52,  27]], dtype=torch.uint8)

TensorDicom uses gray as the cmap default so when you view the image using the show function:

ten_img.show()

<matplotlib.axes._subplots.AxesSubplot at 0x1fb02452748>

However this can be changed to the bone cmap which is better at differentiating different areas of the image.

class TensorDicom(TensorImage): _show_args = {'cmap':'bone'}

ten_img2 = TensorDicom(dimg.pixel_array)
ten_img2.show()

<matplotlib.axes._subplots.AxesSubplot at 0x1fb0261cb48>

Tip: You can easily set default values and styling by using matplotlib.rcParams which in this case sets the cmap values to bone

matplotlib.rcParams['image.cmap'] = 'bone'

Class PILDicom

Inherits from PILBase

Opens a DICOM file from path fn or bytes fn and load it as a PIL Image. The DICOM is opened using pydicom.dcmread and accesses the pixel_array and returns the full size of the image.

type(PILDicom.create(img))

fastai2.medical.imaging.PILDicom

PILDicom.create(img)

pixels

Converts the pixel_array into a tensor

pixels(dimg)

tensor([[  2.,   6.,   5.,  ...,   3.,   3.,   2.],
        [  5.,   9.,   8.,  ...,   6.,   5.,   5.],
        [  5.,   9.,   9.,  ...,   6.,   5.,   5.],
        ...,
        [ 49.,  85.,  80.,  ..., 123., 121.,  69.],
        [ 54.,  88.,  81.,  ..., 118., 115.,  70.],
        [ 17.,  48.,  39.,  ...,  46.,  52.,  27.]])

scaled_pixel

pixels scaled by RescaleSlope and RescaleIntercept. The slim SIIM_SMALL dataset does not have RescaleSlope (0028,1053) or RescaleIntercept (0028,1052) in the dataset.

To illustrate the importance of RescaleSlope and RescaleIntercept, I am going to use an image from this dataset which has a:

RescaleIntercept of -1024
RescaleSlope of 1

test_i = 'D:/Datasets/train_test/167.dcm'
test_im = dcmread(test_i)
timg = PILDicom.create(test_i)
timg.show()

<matplotlib.axes._subplots.AxesSubplot at 0x1fb05561788>

We can then plot a histogram to represent the pixel values within the dicom image

px = test_im.pixels.flatten()
plt.hist(px, color='c')

(array([2.97149e+05, 8.66900e+04, 1.60402e+05, 3.18460e+04, 8.06200e+03,
        3.79700e+03, 1.18900e+03, 4.36000e+02, 1.85000e+02, 6.80000e+01]),
 array([   0. ,  430.7,  861.4, 1292.1, 1722.8, 2153.5, 2584.2, 3014.9,
        3445.6, 3876.3, 4307. ], dtype=float32),
 <a list of 10 Patch objects>)

Note: Air has a value of -1000 Hounsfield Units(HUs), water has a value of 0 HUs and depending on the type of bone, bone has values between 300 and 2000 HUs

Note: This dataset also has a Photometric Interpretation of MONOCHROME2 so the low values ie -1000 HUs are displayed as dark and the higher values ie +1000 HUs are displayed as bright

The image above is a slice of the chest cavity that predominantly consists of air(dark areas) and there is not much bone(bright white areas) in the picture but the histogram shows that the bulk of the pixels are distributed around 0 and 1000 which does not correctly represent the tissues densities in the image.

This why RescaleSlope and RescaleIntercept are important and scaled_px uses these values to correctly scale the image so that they represent the correct tissue densities.

tensor_dicom_scaled = scaled_px(test_im) #convert into tensor taking RescaleIntercept and RescaleSlope into consideration
plt.hist(tensor_dicom_scaled.flatten(), color='c')

(array([2.97149e+05, 8.66900e+04, 1.60402e+05, 3.18460e+04, 8.06200e+03,
        3.79700e+03, 1.18900e+03, 4.36000e+02, 1.85000e+02, 6.80000e+01]),
 array([-1024. ,  -593.3,  -162.6,   268.1,   698.8,  1129.5,  1560.2,
         1990.9,  2421.6,  2852.3,  3283. ], dtype=float32),
 <a list of 10 Patch objects>)

>> Side Note: Pixel Distribution

Having well scaled inputs is really important in getting good results from neural net training ref. This means having a normal or uniform distribution. The pixels in the DICOM image do not show a uniform distribution

#switching back to the SIIM dataset
px = dimg.pixels.flatten()
plt.hist(px, bins=50, color='g');

In this case the image is showing multimodal distribution(having more than 2 peaks). Another case could be where the distribution is bimodal(having 2 distinct peaks). The functions below provide a means of splitting the range of pixel values into groups so that each group has an equal number of pixels

array_freqhist_bins

This is a numpy based function to split the range of pixel values into groups, such that each group has around the same number of pixels

Note: array_freqhist_bins requires an array as the input and the default is set to 100 bins or groups

arry = array_freqhist_bins(dimg.pixel_array)
arry

array([  3,   4,   5,   6,   7,   8,   9,  10,  11,  13,  16,  21,  26,
        31,  38,  45,  52,  57,  62,  65,  68,  70,  72,  74,  76,  78,
        79,  81,  82,  83,  84,  86,  87,  88,  89,  90,  91,  92,  93,
        94,  95,  96,  97,  98,  99, 100, 101, 102, 103, 104, 105, 106,
       107, 108, 109, 110, 111, 112, 113, 115, 116, 117, 119, 120, 122,
       123, 124, 126, 127, 129, 131, 132, 134, 136, 138, 140, 143, 145,
       148, 151, 155, 158, 161, 163, 165, 167, 169, 171, 173, 175, 177,
       179, 180, 182, 184, 186, 189, 192, 197, 203], dtype=uint8)

Tensor.freqhist_bins

Splits the range of pixel values into groups, such that each group has around the same number of pixels where the input is a tensor. You can create a tensor of the image by using pixels

tensor_dicom = pixels(dimg)
tensor_dicom

tensor([[  2.,   6.,   5.,  ...,   3.,   3.,   2.],
        [  5.,   9.,   8.,  ...,   6.,   5.,   5.],
        [  5.,   9.,   9.,  ...,   6.,   5.,   5.],
        ...,
        [ 49.,  85.,  80.,  ..., 123., 121.,  69.],
        [ 54.,  88.,  81.,  ..., 118., 115.,  70.],
        [ 17.,  48.,  39.,  ...,  46.,  52.,  27.]])

You can then specify the number of bins or groups you want to split the pixels into, the default is 100

t_bin = tensor_dicom.freqhist_bins(n_bins=100)
t_bin

tensor([  3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,  13.,  16.,  21.,
         26.,  31.,  38.,  45.,  52.,  57.,  62.,  65.,  68.,  70.,  72.,  74.,
         76.,  78.,  79.,  81.,  82.,  83.,  84.,  86.,  87.,  88.,  89.,  90.,
         91.,  92.,  93.,  94.,  95.,  96.,  97.,  98.,  99., 100., 101., 102.,
        103., 104., 105., 106., 107., 108., 109., 110., 111., 112., 113., 115.,
        116., 117., 119., 120., 122., 123., 124., 126., 127., 129., 131., 132.,
        134., 136., 138., 140., 143., 145., 148., 151., 155., 158., 161., 163.,
        165., 167., 169., 171., 173., 175., 177., 179., 180., 182., 184., 186.,
        189., 192., 197., 203.])

Note: freqhist_bins splits the pixels into bins, the number of bins set by n_bins. So for the above example:

- freqhist_bins flattens out the image tensor (in this case 1024 by 1024 into a flattened tensor of size 1048576 (1024*1024)

- setting `n_bins` to 1 for example means it will be split into 3 bins (the beginning, the end and the number of bins specified by `n_bins`
- each bin is then scaled to values between 0 and 255 (in this case the bulk of pixels are grouped at 3, 103 and 203

with n_bin set to 1

t_bin = tensor_dicom.freqhist_bins(n_bins=1)
plt.hist(t_bin, bins=t_bin)

(array([1., 2.]),
 array([  3., 103., 203.], dtype=float32),
 <a list of 2 Patch objects>)

plt.hist(t_bin, bins=t_bin, color='c'); plt.plot(t_bin, torch.linspace(0,1,len(t_bin)));dimg.show(t_bin)

with n_bins of 100

t_bin = tensor_dicom.freqhist_bins(n_bins=100)
plt.hist(t_bin, bins=t_bin)

(array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 2.]),
 array([  3.,   4.,   5.,   6.,   7.,   8.,   9.,  10.,  11.,  13.,  16.,
         21.,  26.,  31.,  38.,  45.,  52.,  57.,  62.,  65.,  68.,  70.,
         72.,  74.,  76.,  78.,  79.,  81.,  82.,  83.,  84.,  86.,  87.,
         88.,  89.,  90.,  91.,  92.,  93.,  94.,  95.,  96.,  97.,  98.,
         99., 100., 101., 102., 103., 104., 105., 106., 107., 108., 109.,
        110., 111., 112., 113., 115., 116., 117., 119., 120., 122., 123.,
        124., 126., 127., 129., 131., 132., 134., 136., 138., 140., 143.,
        145., 148., 151., 155., 158., 161., 163., 165., 167., 169., 171.,
        173., 175., 177., 179., 180., 182., 184., 186., 189., 192., 197.,
        203.], dtype=float32),
 <a list of 99 Patch objects>)

plt.hist(t_bin, bins=t_bin, color='c'); plt.plot(t_bin, torch.linspace(0,1,len(t_bin)));dimg.show(t_bin)

Note: You can notice that the quality of the images is different between the 2 sets

Tensor.hist_scaled_pt

Tensor.hist_scaled

A way to scale a tensor of pixels evenly using freqhist_bins to values between 0 and 1.

tensor_dicom

tensor([[  2.,   6.,   5.,  ...,   3.,   3.,   2.],
        [  5.,   9.,   8.,  ...,   6.,   5.,   5.],
        [  5.,   9.,   9.,  ...,   6.,   5.,   5.],
        ...,
        [ 49.,  85.,  80.,  ..., 123., 121.,  69.],
        [ 54.,  88.,  81.,  ..., 118., 115.,  70.],
        [ 17.,  48.,  39.,  ...,  46.,  52.,  27.]])

The tensor has values between 0 and 255

#Run to view - commented out to reduce file size
#plt.hist(tensor_dicom, bins=100)

The above commented out code displays a histogram of pixel values which range from 0 to 255

Using hist_scaled gets values now scaled between 0 and 1

tensor_hists = tensor_dicom.hist_scaled()
tensor_hists

tensor([[0.0000, 0.0303, 0.0202,  ..., 0.0000, 0.0000, 0.0000],
        [0.0202, 0.0606, 0.0505,  ..., 0.0303, 0.0202, 0.0202],
        [0.0202, 0.0606, 0.0606,  ..., 0.0303, 0.0202, 0.0202],
        ...,
        [0.1573, 0.3081, 0.2677,  ..., 0.6566, 0.6414, 0.2071],
        [0.1657, 0.3333, 0.2727,  ..., 0.6212, 0.5960, 0.2121],
        [0.1030, 0.1558, 0.1429,  ..., 0.1530, 0.1616, 0.1232]])

#plotting the scaled histogram
#plt.hist(tensor_hists, bins=100)

Scaled histogram now has pixel values ranging from 0 to 1

Dataset.hist_scaled

Similar to Tensor.hist_scaled except that the input here is a DcmDataset and you can set the minimum and maximum pixel values

dimg.pixel_array

array([[  2,   6,   5, ...,   3,   3,   2],
       [  5,   9,   8, ...,   6,   5,   5],
       [  5,   9,   9, ...,   6,   5,   5],
       ...,
       [ 49,  85,  80, ..., 123, 121,  69],
       [ 54,  88,  81, ..., 118, 115,  70],
       [ 17,  48,  39, ...,  46,  52,  27]], dtype=uint8)

data_scaled = dimg.hist_scaled(min_px=1, max_px=20)
data_scaled

tensor([[0.0000, 0.2727, 0.1818,  ..., 0.0000, 0.0000, 0.0000],
        [0.1818, 0.5455, 0.4545,  ..., 0.2727, 0.1818, 0.1818],
        [0.1818, 0.5455, 0.5455,  ..., 0.2727, 0.1818, 0.1818],
        ...,
        [1.0000, 1.0000, 1.0000,  ..., 1.0000, 1.0000, 1.0000],
        [1.0000, 1.0000, 1.0000,  ..., 1.0000, 1.0000, 1.0000],
        [0.9318, 1.0000, 1.0000,  ..., 1.0000, 1.0000, 1.0000]])

#Using min_px = 1 and max_px = 20
plt.imshow(data_scaled, cmap=plt.cm.bone);

#Using min_px = 100 and max_px = 244
data_scaled2 = dimg.hist_scaled(min_px=100, max_px=244)
plt.imshow(data_scaled2, cmap=plt.cm.bone)

<matplotlib.image.AxesImage at 0x1fb0739a108>

>> Side Note: Windowing

DICOM images can contain a high amount of voxel values and windowing can be thought of as a means of manipulating these values in order to change the apperance of the image so particular structures are highlighted. A window has 2 values:

l = window level or center aka brightness

w = window width or range aka contrast

Example: from here

Brain Matter window

l = 40 (window center) w = 80 (window width)

Voxels displayed range from 0 to 80

Calculating voxel values:

lowest_visible_value = window_center - window_width / 2
highest_visible_value = window_center + window_width / 2

(lowest_visible_value = 40 - (80/2), highest_visible_value = 40 + (80/2))

Note: Hence all values >80 will be white and all values below 0 are black.

windowed

Takes 2 values w and l

fastai conveniently provides a range of window width and centers (dicom_windows) for viewing common body areas:

brain=(80,40),
subdural=(254,100),
stroke=(8,32),
brain_bone=(2800,600),
brain_soft=(375,40),
lungs=(1500,-600),
mediastinum=(350,50),
abdomen_soft=(400,50),
liver=(150,30),
spine_soft=(250,50),
spine_bone=(1800,400)

plt.imshow(dimg.windowed(w=1500, l=-600), cmap=plt.cm.bone)

<matplotlib.image.AxesImage at 0x1fb03e4f288>

The above is the same as below using dicom_windows.lungs

plt.imshow(dimg.windowed(*dicom_windows.lungs), cmap=plt.cm.bone)

<matplotlib.image.AxesImage at 0x1fb073b6748>

Windowing example

Example of how windowing works

img = items[10]
dimg = dcmread(img)

Convert a DICOM image into tensors

tensor_dicom = pixels(dimg)
tensor_dicom

tensor([[  2.,   6.,   5.,  ...,   3.,   3.,   2.],
        [  5.,   9.,   8.,  ...,   6.,   5.,   5.],
        [  5.,   9.,   9.,  ...,   6.,   5.,   5.],
        ...,
        [ 49.,  85.,  80.,  ..., 123., 121.,  69.],
        [ 54.,  88.,  81.,  ..., 118., 115.,  70.],
        [ 17.,  48.,  39.,  ...,  46.,  52.,  27.]])

View a portion of the image

portion = tensor(tensor_dicom)[200:440,600:840]
portion.shape

torch.Size([240, 240])

df = pd.DataFrame(portion)

#uncomment to run on locally
#df.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')

The dataframe created by df looks like this:

You can create your own window parameters, here we create one named test_window

dicom_windows = types.SimpleNamespace(
    test_window=(50,60)
)

We can apply the test_window window to portion image

r = portion.windowed(w=50, l=60)

#This will produce a Dataframe of each pixel whether 'true' or 'false'
#uncomment to run locally

test_e = portion.mask_from_blur(dicom_windows.test_window, thresh=0.05, remove_max=True)
#dfe = pd.DataFrame(test_e)
#dfe.style.set_properties(**{'font-size':'6pt'}).background_gradient('Greys')

imk = [portion,r, test_e]
show_images(imk, titles=('orig', 'windowed', 'mask'), figsize=(15,15))

using show_images the images appear green because show_images and show_image by default use the 'viridis' colormap and PILBase follows the same default colormap. However we had earlier changed the Matplotlib.rcParams to bone

show

Inherits from show_image and is customized for displaying DICOM images

dimg.show(scale=True, cmap=plt.cm.bone, min_px=-1100, max_px=None, ax=None, figsize=(6,6), title='Test', ctx=None)

Jeremy mentioned in this article about using a 'rainbow colormap' to fully utilize our computer's ability to display color

dimg.show(cmap=plt.cm.gist_ncar, figsize=(6,6))

Class TensorCTScan

This class returns TensorImageBW

Class PILCTScan

Inherits from PILBase

pct_in_window

Gets the % of pixels within a window w, l

For example using lungs as the dicom_window (1500,-600) we see that 80% of the pixels are within the window

dimg.pct_in_window(*dicom_windows.lungs)

0.7996940612792969

What about the test_window that was created:

dimg.pct_in_window(*dicom_windows.test_window)

0.1637105941772461

uniform_blur2d

A way of blurring the image uniformarly taking in 2 inputs: the input image and the scale of blurring

ims = dimg.hist_scaled(), uniform_blur2d(dimg.hist_scaled(),20), uniform_blur2d(dimg.hist_scaled(),50)
show_images(ims, titles=('original', 'blurred 20', 'blurred 50'), figsize=(15,15))

gauss_blur2d

Uses gaussian_blur2d kornia filter

gims = dimg.hist_scaled(), gauss_blur2d(dimg.hist_scaled(),20), uniform_blur2d(dimg.hist_scaled(),50)
show_images(gims, titles=('original', 'gauss_blurred 20', 'gauss_blurred 50'), figsize=(15,15))

As explained in this notebook Jeremy uses these methods in cleaning the data in order to only retain information that is relevant. For example you use gauss_blur2d to blur the image and then select the areas that are bright

px = dimg.windowed(w=1100, l=-500)
show_image(px, figsize=(4,4));

blurred = gauss_blur2d(px, 125)
show_image(blurred, figsize=(4,4));

show_image(blurred>1, figsize=(4,4));

mask_from_blur

Function to create a mask from the blurred image that takes in the input and window, sigma and threshold values

#switching back to this image which allows the use of `scaled_px`
test_i = 'D:/Datasets/train_test/167.dcm'
test_im = dcmread(test_i)
timg = PILDicom.create(test_i)
timg.show()

<matplotlib.axes._subplots.AxesSubplot at 0x1fb08f2cf88>

mask = test_im.mask_from_blur(dicom_windows.lungs, sigma=0.1, thresh=0.75, remove_max=False)
wind = test_im.windowed(*dicom_windows.lungs)

_,ax = subplots(1,2)
show_image(wind, ax=ax[0])
show_image(mask, alpha=0.5, cmap=plt.cm.Reds, ax=ax[1]);

mask2bbox

bbs = mask2bbox(mask)
lo,hi = bbs
show_image(wind[lo[0]:hi[0],lo[1]:hi[1]]);

crop_resize

Dataset.to_nchan

show_images(dimg.to_nchan([dicom_windows.brain,dicom_windows.subdural,dicom_windows.abdomen_soft]))

show_images(dimg.to_nchan([dicom_windows.brain]))

Dataset.to_3chan

Tensor.save_jpg

Save a tensor into a .jpg with specified windows

tensor_dicom = pixels(dimg)
tensor_dicom.save_jpg(path=(pneumothorax_source/f'train/01tensor.jpg'), wins=[dicom_windows.lungs, dicom_windows.subdural])

show_image(Image.open(pneumothorax_source/f'train/01tensor.jpg'))

<matplotlib.axes._subplots.AxesSubplot at 0x239905644c8>

Dataset.save_jpg

Save a DICOM image into a .jpg

dimg.save_jpg(path=(pneumothorax_source/f'train/01dicom.jpg'), wins=[dicom_windows.lungs, dicom_windows.lungs])

show_image(Image.open(pneumothorax_source/f'train/01dicom.jpg'))

<matplotlib.axes._subplots.AxesSubplot at 0x2398cd85f08>

Dataset.set_pixels

Dataset.zoom

Zoom's image by specifying pixel ratio

dimg2 = dcmread(items[11])
dimg2.zoom(0.1)
dimg2.show()

dimg2.zoom(0.5)
dimg2.show()

Dataset.zoom_to

dimg2 = dcmread(items[11])
dimg2.shape, dimg2.show()

((1024, 1024), None)

dimg2.zoom_to(90)
dimg2.shape, dimg2.show()

((90, 90), None)

epoch	train_loss	valid_loss	accuracy	time
0	1.252483	0.934061	0.620000	00:05
1	1.278853	1.002197	0.460000	00:05
2	1.280225	0.949744	0.540000	00:05

epoch	train_loss	valid_loss	accuracy	time
0	0.678739	3.618579	0.360000	00:10
1	0.680920	0.631048	0.740000	00:10
2	0.661906	0.582651	0.720000	00:10

	file	PatientID	Age	Sex	Rows	Cols
0	C:\Users\avird\.fastai\data\siim_small\sm\No Pneumothorax\000000 - Copy.dcm	16d7f894-55d7-4d95-8957-d18987f0e981	62	M	1024	1024
1	C:\Users\avird\.fastai\data\siim_small\sm\No Pneumothorax\000000.dcm	16d7f894-55d7-4d95-8957-d18987f0e981	62	M	1024	1024
2	C:\Users\avird\.fastai\data\siim_small\sm\No Pneumothorax\000002.dcm	850ddeb3-73ac-45e0-96bf-7d275bc83782	52	F	1024	1024
3	C:\Users\avird\.fastai\data\siim_small\sm\No Pneumothorax\000005.dcm	e0fd6161-2b8d-4757-96bc-6cf620a993d5	65	F	1024	1024
4	C:\Users\avird\.fastai\data\siim_small\sm\No Pneumothorax\000006 - Copy.dcm	99171908-3665-48e8-82c8-66d0098ce209	52	F	1024	1024

Medical Imaging

Creating Bounding Boxes from Class Activation Maps

Load the Data and Create DataBlock

Generating the heatmaps

How do you generate the bounding boxes?

get_maps

get_boxes

Fmi: Additional Functionality For Fastai’s Medical Imaging Module

About

Installation

Tutorials

A Matter of Grayscale: Understanding Dicom Windows

A Matter of Grayscale

Windowing

Fastai's in-built windows

How does windowing work?

Significance of changing window settings

Various window ranges

Lung Window

Mediastinum Window

PE Window

References

Selective Mask

Goal:

Original Mask

Selective Mask

Motivation

Dataset

Selective Mask ✔

Comparing masks from different isup grades

DataBlock

Using Original Mask

Using Selective Mask

Next Steps

Understanding Photometric Interpretation

Monochrome

YBR_FULL_422

Getting to know DICOMS

DICOM Datasets

Loading DICOMs which have 1 frame per file

Loading DICOMs which have multiple frames per file ✔

Saving files from multiple frames

Viewing DICOM tag values

Understanding Tissue Densities

Bins

fin

Starting with Medical Imaging

Goal:

Load the Data

Viewing the Data

View Dicom

Create a Dataframe of all the tags in the header section

EDA

Modelling

Patient Overlap

DicomSplit ✔

Sampling

DataSplit ✔

Augmentations

Training

show_results

Model Evaluation

Accuracy

False Positive & False Negative

Sensitivity & Specificity

PPV and NPV

Confusion Matrix

Calculating Accuracy

fin

Medical Imaging using Fastai

Patching

get_dicom_files

dcmread

Understanding Dicoms

Class TensorDicom

Class PILDicom

pixels

scaled_pixel

>> Side Note: Pixel Distribution

array_freqhist_bins

`Fastai`'s in-built windows

Viewing DICOM `tag` values

Create a Dataframe of all the `tags` in the header section