PyTorch Deep Explainer

[gabrieltseng]

This pull request is the second part of a solution to #191. I would appreciate any feedback!

Here are two notes about this pull request:

1. The deep explainer doesn't quite go back along PyTorch's backpropagation graph.

In PyTorch, this graph is generated using Function objects; these used to have a saved_tensors attribute, which made the input tensors accessible. However, as more and more functions are being written in C++, this is no longer the case. If these saved tensors are exposed once again, it would make sense to use them, especially since function objects can handle backward hooks (which allow the gradients to be manipulated).

What is implemented is a backpropagation along the Module objects, which are a little coarser. As an example, the following network works with the Deep Explainer, because all operations are captured in a module object:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv_layers = nn.Sequential(
            nn.Conv2d(1, 10, kernel_size=5),
            nn.MaxPool2d(2),
            nn.ReLU(),
            nn.Conv2d(10, 20, kernel_size=5),
            nn.Dropout(),
            nn.MaxPool2d(2),
            nn.ReLU(),
        )
        self.fc_layers = nn.Sequential(
            nn.Linear(320, 50),
            nn.ReLU(),
            nn.Dropout(),
            nn.Linear(50, 10),
            nn.Softmax(dim=1)
        )

    def forward(self, x):
        x = self.conv_layers(x)
        x = x.view(-1, 320) # doesn't affect the gradient, so can be outside an nn.Module object
        x = self.fc_layers(x)
        return x

but the following doesn't:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.softmax(x, dim=1)

because the relu activations, the maxpooling and the dropout happen using functions, not modules. I understand that this limits the functionality somewhat, and did spend some time trying to use the Functions instead.

2. The API for this feature is identical to the PyTorch Gradient Explainer; the entire model can be explained by simply passing the model, and if a tuple (model, layer) is passed, then the inputs of that interim layer will be explained.

[slundberg]

Thanks!! This is great. The limitation about functions will probably cause trouble for people, but this is much better than nothing. We can figure out later how to support functions (potentially supported by later releases of pytorch).

I am on travel right now and will be for the next 11 days and won't have a chance to carefully review this until I am back. But I'll look it over to get it merged as soon as I can.

[slundberg]

Thanks again! Merging this now :) ...if they re-expose saved_tensors then we can look at that then. I expect there will be some corner case stuff we run into, but that can be fixed when we find it.