Here our problem statements transition from wishful thinking to something that is realistic, measurable, and scalable (Note that we didn't define any metrics without context. For example we could have stated we wanted to 'reduce costs by 10%.' However, by choosing to exclude those abstract metrics, we reduce biases going into our project, such as cognitive dissonance, availability heuristic, and affect heuristic [see more]. By allowing ourselves space for freedom and creativity, even with a strongly defined problem, we protect ourself from the rigidity of metric-driven analysis. This is not to say that problems cannot be framed in context of quantities or statistics, however, it is simply out of the scope of this blog to go over the best practices associated with said problem set. We may frame the problem so that we make no assumptions about the solution, but that allows us to create a bounded decision space in which our optimal solution exists. Importantly, this lets us maintain a skeptics mindset throughout this process; from construction of the problem to the creation of a solution, do not accept new information blindly. If we look to refute rather than confirm our ideas/opinions/models/etc, we'll be forced to ground our reasoning in reality and help produce applicable results. An example from data science: you obtain 90% model accuracy. Instead of hopping up and down with joy, it is best to say, 'Hmm, I wonder why my model is so good. Is it unreasonable to expect this on my second try? Instead of accepting this blindly, I will work to refute the results. If I cannot, then I will move forward and document what I did to test my results, preventing others from blindly accepting my results.' As you can see, there are mighty parallels between the scientific method and the explainable workflow.
Above we changed a weakly defined problem into a strongly defined one through prototyping. Weakly defined problems are good for producing ideas, strongly defined problems are good for producing results. It is important to document this transition between states to isolate what solutions were derived from what ideas. These in turn can be implemented into best business practices, increased acumen in this domain, and less redundant work.
Algorithm Selection & Hyperparameter Tuning
For the next step in the overarching machine learning process, after we have isolated a problem, we need to look for our data. It's essential to remember that you should never find data, and then ask yourself, 'what problems can I solve with this.' Almost without exception, by doing so you'll introduce 'confirmation bias.' Confirmation bias occurs from the direct influence of desire on beliefs. We want to stay as objective and logically consistent as possible. That is why we formulate our problem before locating data. If you wish to learn more on this thought process, take a look at this great blog: https://yanirseroussi.com/2015/08/24/you-dont-need-a-data-scientist-yet/
Once we've decided on the data we need and we've acquired it, we can finally start the process of choosing our algorithm. In terms of explainability/interpretability, not all algorithms are made equal. The design of some models makes their relationship between inputs and outputs more understandable. We won't compare each and every model, but we want to give you a sample for the purposes of comparison and to understand the relative strengths and weaknesses of some algorithm designs.
Creation, instantiation, and training of a CART decision tree is inherently more interpretable than a XGBoostClassifier; simply by nature of the mathematics behind each model. CART decision trees utilize the GINI index to measure potential information gains or losses at each split in the tree. Those indices as chosen by the model can be visually inspected. We can look at the index and confirm for ourselves the respective values of our features upon which our CART decision tree decided to split. That architecture is the very essence of an interpretable outcome. The XGBoostClassifier, on the other hand, builds predictions on gradient boosting across an ensemble of weak learners (think of them as decision trees which have been pruned down to just a couple of decision nodes), which in turn creates strong learners. The predictions created from this are the result of a process less easily abstracted for human understanding, and therefore this model has its limitations in explainability in down-stream tasks. We're using these algorithms to show there is a very wide range in potential interpretability based on the initial designs of the models themselves.
Furthermore, explainability can be further divided down into model-specific explainability and model-agnostic explainability. Model-specific methods differ between each unique model; because of this it may be best to use model-agnostic methods [LIME}. Model-agnostic methods will provide a general language to be able to work with a wider variety of models. Looking forward, this will also act as an easier framework to remember and to implement in personal projects, academics, jobs or the like.
Explainability is a nascent field, therefore, algorithms and frameworks are still being created and codified to grapple with newly occuring problems. Two popular model-agnostic methods are integrated gradients and shapley values. There are some limitations associated with these methods, which we will discuss further in the blog. When looking at what model to use, any model that uses gradient descent is differentiable, therefore, integrated gradients may be better. This will include neural networks, logistic regression, support vector machines, and other algorithms that make use of gradients. If the models are non-differentiable then shapley values will work. This will include trees, i.e. boosted trees and random forests.
What is the difference between interpretability and explainability?
Interpretability is the ability to see a models parameters and equations in a static manner. An example of this in computer vision would be obtaining the prediction for a specific observation, and statically analyzing the agreement between your results and expectations, which can be captured via notes.
Explainability is the ability to iteratively diagnose a model in a dynamic manner. When DARPA came out with their XAI challenge in 2016 they had 3 main objectives: understand how to produce more explainable models, understand how to design the explanation interface, and understand how to build requirements for effective explanations. To understand explainability we can look empirically at how we learn. We might use counterfactuals, comparisons, context through stories, point out errors, or look for causality [correlation != causation].
How will this affect our workflow?
We add explainability processes in an attempt to limit, remove and identify bias, to implement monitoring, and to implementing a design plan to add a human in the loop. Consider the counterfactual scenario where we ignore the impact of bias in our models. In it, if our blackbox models predict outcomes with systematic bias included as input to our data set, we'll taint our predictions from the very beginning and have a difficult time explaining, or justifying, our outputs. They are by definition less explainable. Instead, if we account for bias in input data by adding explainability processes to our workflow, we can mitigate the effect of bias on our predictions. Our steps are as follows, presuming we've attained and cleaned our data: Look at the business understanding on the data bias through risk assessment. Define features, select a model and parameters, and define the scoring metrics along with calculating the accuracy. Once this is done, we rinse and repeat as needed. The next added step is to look at the business logic as well as the modeling, metric and feature understanding along with risk management. Persist the model and the code, and then predict using transformations and the trained model. The last added step is evaluate the requirements for additional infrastructure needed for monitoring, and to add process design plans to include human interactions in the loop. Overall this new workflow will add more interaction from key stakeholders as well as create greater opportunity for data science to explore feature importance, accuracy and error. We realize we're asking a significant investment for the sake of adding explainable elements to your data science processes, but only then can you feel comfortable with the justifications your models' predictions will demand.
Gradient Magnitudes with Facebook Captum
# Pytorch with Facebook's Captum with an example of Integrated Gradients.
# Code from Pytorch tutorial with some changes: full example: https://captum.ai/tutorials/CIFAR_TorchVision_Interpret
# !pip install captum
import PIL
"""Pillow 7.0.0 is deprecated for PILLOW_VERSION.py in Setup folder, use version 6.2.1 with
{pip install pillow==6.2.1}"""
import numpy as np
import torch.nn as nn
import torch, torchvision
import torch.optim as optim
from torchvision import models
import torch.nn.functional as F
import torchvision.transforms as transforms
import torchvision.transforms.functional as TF
from captum.attr import IntegratedGradients
from captum.attr import visualization as viz
# install cuda: https://developer.nvidia.com/cuda-downloadstransform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool1 = nn.MaxPool2d(2, 2)
self.pool2 = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
self.fc1 = nn.Linear(16 * 5 * 5, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 10)
self.relu1 = nn.ReLU()
self.relu2 = nn.ReLU()
self.relu3 = nn.ReLU()
self.relu4 = nn.ReLU()
def forward(self, x):
x = self.pool1(self.relu1(self.conv1(x)))
x = self.pool2(self.relu2(self.conv2(x)))
x = x.view(-1, 16 * 5 * 5)
x = self.relu3(self.fc1(x))
x = self.relu4(self.fc2(x))
x = self.fc3(x)
return x
net = Net()criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
dataiter = iter(testloader)
images, labels = dataiter.next()
outputs = net(images)
_, predicted = torch.max(outputs, 1)
ind = 3
input = images[ind].unsqueeze(0)
input.requires_grad = True
net.eval()def attribute_image_features(algorithm, input, **kwargs):
net.zero_grad()
tensor_attributions = algorithm.attribute(input,target=labels[ind],**kwargs)
return tensor_attributions
ig = IntegratedGradients(net)
attr_ig, delta = attribute_image_features(ig, input, baselines=input * 0, return_convergence_delta=True)
attr_ig = np.transpose(attr_ig.squeeze().cpu().detach().numpy(), (1, 2, 0))
print('Approximation delta: ', abs(delta)) # The lower the abs error the better the approximation.
Delta is the difference between the total approximated and true integrated gradients. Using delta we can preview the underpinnings of the model and how the predicted and the actual compare to one another. We could preview a whole image as well with one output to one pixel. Although we can preview the error of each given pixel or feature it doesn’t necessarily provide us with insight into what exactly we’re looking at.
original_image = np.transpose((images[ind].cpu().detach().numpy()) + 0.5, (1, 2, 0))
_ = viz.visualize_image_attr(None, original_image, method="original_image", title="Original Image")
_ = viz.visualize_image_attr(attr_ig, original_image, method="blended_heat_map",sign="all", show_colorbar=True, title="Overlay of Integrated Gradients")
The code above produces the following outputs:
The image on the top is the original image of a ship. The image on the bottom is the overlay of gradient magnitudes showing what attributes contribute to the overall prediction of a ship. Integrated gradients are being used just as before. Although the image is really low resolution we can make out a set of pixels that were used for the prediction. By viewing certain pixels or superpixels we can see general behaviors of how the model is making predictions.
Explainability is a developing topic, especially when using images. A lot of scenarios that are being used involve having deep domain knowledge. This is where high level semantic feature explanations will come more into play, and are currently being worked on.
Image Explainability with LIME
# Lime with ImageNet and Pytorch. Code pulled from 'github.com/marcotcr/', a lime library.
import numpy as np
import torchvision
import torch.nn as nn
import torch, os, json
import torch.nn.functional as F
import matplotlib.pyplot as plt
from PIL import Image
from torch.autograd import Variable
from torchvision import models, transforms#to run you will need the imagenet_class_index.json and the dogs.png image from ImageNet.
#available at: http://www.image-net.org/
imagenet_data = torchvision.datasets.ImageFolder(root = '/')
data_loader = torch.utils.data.DataLoader(imagenet_data, batch_size=4, shuffle=True)
# convert image to tensor and also apply whitening as used by pretrained model.
# Resize and take the center part of image to what our model expects.
def get_input_transform():
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
transf = transforms.Compose([transforms.Resize((256, 256)), transforms.CenterCrop(224), transforms.ToTensor(), normalize])
return transf
def get_input_tensors(img):
transf = get_input_transform()
return transf(img).unsqueeze(0) # unsqueeze converts single image to batch of 1
model = models.inception_v3(pretrained=True) # load ResNet50
# Load label texts for ImageNet predictions so we know what model is predicting
idx2label, cls2label, cls2idx = [], {}, {}
with open(os.path.abspath('./imgnet_data/imagenet_class_index.json'), 'r') as read_file:
class_idx = json.load(read_file)
idx2label = [class_idx[str(k)][1] for k in range(len(class_idx))]
cls2label = {class_idx[str(k)][0]: class_idx[str(k)][1] for k in range(len(class_idx))}
cls2idx = {class_idx[str(k)][0]: k for k in range(len(class_idx))}
#grab image and make as input
def get_image(path):
with open(os.path.abspath(path), 'rb') as f:
with Image.open(f) as img:
return img.convert('RGB')img = get_image('./imgnet_data/dogs.png')
# Get prediction for image.
img_t = get_input_tensors(img)
model.eval()
logits = model(img_t)
# Predictions are logits; pass through softmax to get probabilities and class labels for top 5 predictions.
probs = F.softmax(logits, dim=1)
probs5 = probs.topk(5)
tuple((idx2label[c], p) for p, c in zip(probs5[0][0].detach().numpy(), probs5[1][0].detach().numpy()))
The code above outputs the following probabilities for an image of a dog passed to the network:
(('Bernese_mountain_dog', 0.93593013),
('EntleBucher', 0.038447786),
('Appenzeller', 0.02375631),
('Greater_Swiss_Mountain_dog', 0.0018181783),
('Gordon_setter', 9.113291e-06))
from lime import lime_image
from skimage.segmentation import mark_boundariesdef get_pil_transform():
transf = transforms.Compose([transforms.Resize((256, 256)),transforms.CenterCrop(224)])
return transf
def get_preprocess_transform():
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
transf = transforms.Compose([transforms.ToTensor(), normalize])
return transfpill_transf = get_pil_transform()
preprocess_transform = get_preprocess_transform()def batch_predict(images):
model.eval()
batch = torch.stack(tuple(preprocess_transform(i) for i in images), dim=0)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
batch = batch.to(device)
logits = model(batch)
probs = F.softmax(logits, dim=1)
return probs.detach().cpu().numpy()test_pred = batch_predict([pill_transf(img)])
test_pred.squeeze().argmax()
explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(np.array(pill_transf(img)),
batch_predict, # classification function
top_labels=5,
hide_color=0,
num_samples=10000) # number of images that will be sent to classification function; impacts resulting image boundaries for next celltemp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=True, num_features=5, hide_rest=True)
img_boundry1 = mark_boundaries(temp/255.0, mask)
plt.imshow(img_boundry1)
Explainability in workflows helps improve or better define the following: debugging, stakeholder trust, insight in specific outputs, insight in high level concepts of models, and an overall understanding of how a model works. Let’s look further at the overall understanding. This can be further broken down into debugging, monitoring, transparency, and auditing. When looking at model performance, a common area of struggle is debugging, especially with more complex models such as random forests or neural nets.
By utilizing explainability we can see causes for why the model performs poorly on specific inputs. This naturally leads to better feature engineering and knowing why we are dropping certain features due to things such as redundancy. Let’s look at this in a little more detail. One way to look at this is that we want to be able to attribute a model’s prediction to its feature inputs. For example, we want to know why the prediction of hypoxemia (low oxygen in the blood) for an anesthetized patient is being predicted, not just that our model is predicting a 90% accuracy. Otherwise we aren’t sure which of the 7 different knobs the anesthesiologist needs to turn. We call this the attribution problem. One such attribution method is gradient based. This takes feature values gradients ( xi 𝜕y/𝜕x ). The gradient captures sensitivity of output with respect to the feature; remember partial derivatives for this.
Github repo for codebase: https://github.com/decentdilettante/XAI/blob/master/explainability.v.1.2.ipynb
Co-author: Vikas Ramachandra
Sign up for a Watson Studio Desktop trial, here.