Explaining Black-Box Models With LIME

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. In this session, we take the H2O models we’ve developed and show you how to explain localized prediction results using a special technique called LIME (Local Interpretable Model-Agnostic Explanations).

In this short session, you will learn:

How to make an explainer using the lime() function for single and multiple employees
How to make an explanation using the explain() function for single and multiple employees
How to visualize model explanations using plot_features() (one or a small number of observations) and plot_explanations() (for many observations)

Theory Input

Machine learning models are often considered “black boxes” due to their complex inner-workings. More advanced ML models such as random forests, gradient boosting machines (GBM), artificial neural networks (ANN), among others are typically more accurate for predicting nonlinear, faint, or rare phenomena. Unfortunately, more accuracy often comes at the expense of interpretability, and interpretability is crucial for business adoption, model documentation, regulatory oversight, and human acceptance and trust. Luckily, several advancements have been made to aid in interpreting ML models.

Moreover, it’s often important to understand the ML model that you’ve trained on a global scale, and also to zoom into local regions of your data or your predictions and derive local explanations. Global interpretations help us understand the inputs and their entire modeled relationship with the prediction target, but global interpretations can be highly approximate in some cases. Local interpretations help us understand model predictions for a single row of data or a group of similar rows.

This session demonstrates how to use the lime package (install.packages("lime")) to perform local interpretations of ML models. This session will not focus on the theoretical and mathematical underpinnings but, rather, on the practical application of using lime.

The following video gives a good explanation of lime:

Business case

We are still working with our attrition data. Let’s load a H2O model from the alst time.

# LIME FEATURE EXPLANATION ----

# 1. Setup ----

# Load Libraries 

library(h2o)
library(recipes)
library(readxl)
library(tidyverse)
library(tidyquant)
library(lime)

# Load Data
employee_attrition_tbl <- read_csv("datasets-1067-1925-WA_Fn-UseC_-HR-Employee-Attrition.csv")
definitions_raw_tbl    <- read_excel("data_definitions.xlsx", sheet = 1, col_names = FALSE)

# Processing Pipeline
source("00_Scripts/data_processing_pipeline.R")

employee_attrition_readable_tbl <- process_hr_data_readable(employee_attrition_tbl, definitions_raw_tbl)

# Split into test and train
set.seed(seed = 1113)
split_obj <- rsample::initial_split(employee_attrition_readable_tbl, prop = 0.85)

# Assign training and test data
train_readable_tbl <- training(split_obj)
test_readable_tbl  <- testing(split_obj)

# ML Preprocessing Recipe 
recipe_obj <- recipe(Attrition ~ ., data = train_readable_tbl) %>%
                step_zv(all_predictors()) %>%
                step_mutate_at(c("JobLevel", "StockOptionLevel"), fn = as.factor) %>% 
                prep()

recipe_obj

train_tbl <- bake(recipe_obj, new_data = train_readable_tbl)
test_tbl  <- bake(recipe_obj, new_data = test_readable_tbl)

# 2. Models ----

h2o.init()

automl_leader <- h2o.loadModel("04_Modeling/h20_models/StackedEnsemble_BestOfFamily_AutoML_20200903_144246")
automl_leader

Let’s move to the LIME section. To use lime we need our predictions.

h2o.predict() generates predictions using an H2O model and newdata as an H2O Frame.
bind_cols() binds two data frames together column-wise (makes wider).

# 3. LIME ----

# 3.1 Making Predictions ----

predictions_tbl <- automl_leader %>% 
    h2o.predict(newdata = as.h2o(test_tbl)) %>%
    as.tibble() %>%
    bind_cols(
        test_tbl %>%
            select(Attrition, EmployeeNumber)
    )

predictions_tbl
## # A tibble: 220 x 5
##    predict    No    Yes Attrition EmployeeNumber
##    <fct>   <dbl>  <dbl> <fct>              <dbl>
##  1 Yes     0.363 0.637  Yes                    1
##  2 No      0.863 0.137  No                    15
##  3 No      0.963 0.0374 No                    20
##  4 No      0.868 0.132  No                    21
##  5 No      0.952 0.0483 No                    38
##  6 No      0.808 0.192  No                    49
##  7 No      0.930 0.0696 No                    54
##  8 Yes     0.559 0.441  No                    61
##  9 Yes     0.412 0.588  No                    62
## 10 No      0.936 0.0640 No                    70
## # … with 210 more rows

Let’s investigate the 1st employee, that did indeed leave the company:

slice() selects specific rows by rownumber.

test_tbl %>%
    slice(1) %>%
    glimpse()

Lime for single explanation, Part 1:

LIME is used to determine which features contribute to the prediction (& by how much) for a single observation (i.e. local). h2o, keras & caret R packages have been integrated into lime. If you ever need to use an unintegrated package, you can do so by creating special functions: model_type(), predict_model(). Using Lime is is a 2 steps process:

Build an explainer with lime() (“recipe” for creating an explanation. It contains the ML model & feature distributions (bins) for the training data.)
Create an explanation with explain()

lime() creates an “explainer” from the training data & the model object. The returned object contains the ML model and the feature distributions for the training data.
First we have to remove the target feature: The H2O model does not use the Attrition column within the prediction set.
Use bin_continuous to bin the features. It makes it easy to detect what causes the continuous feature to have a high feature weight in the explanation.
Use n_bins to tell how many bins you want. Usually 4 or 5 is sufficient to describe a continuous feature.
Use quantile_bins to tell how to distribute observations with the bins. If TRUE, cuts will be selected to evenly distribute the total observations within each of the bins.

# 3.2 Single Explanation ----

explainer <- train_tbl %>%
    select(-Attrition) %>%
    lime(
        model           = automl_leader,
        bin_continuous  = TRUE,
        n_bins          = 4,
        quantile_bins   = TRUE
    )

explainer

What is inside the explainer object?

1st argument is model. Passes model to the explain() function.
2nd set of arguments are selections for:
- bin_continuous
- n_bins
- quantile_bins
- use_density
- bin_cuts: stores the cuts for every feature that is “continuous” (numeric in feature_type)

Lime For Single Explanation, Part 2: Making an explaination with explain()

LIME Algorithm 6 Steps:

Given an observation, permute it to create replicated feature data with slight value modifications.
Compute similarity distance measure between original observation and permuted observations.
Apply selected machine learnign model to predict outcomes of permuted data.
Select m number of features to best describe predicted outcomes.
Fit a simple model to the permuted data, explaining the complex model outcome with m features from the permuted data weighted by its similarity to the original observation.
Use the resulting feature weights to explain local behaviour.

The data argument must match the format that the model requires to predict. Since h2o.predict() requires “x” to be without the target variable, we must remove. Use lime::explain() since explain() is a common function used in other packages. You will get errors if the incorrect explain() function is used.

kernel_width: Affects the lime linear model fit (R-squared value) and therefore should be tuned to make sure you get the best explanations.

?lime::explain

explanation <- test_tbl %>%
    slice(1) %>%
    select(-Attrition) %>%
    lime::explain(
    
        # Pass our explainer object
        explainer = explainer,
        # Because it is a binary classification model: 1
        n_labels   = 1,
        # number of features to be returned
        n_features = 8,
        # number of localized linear models
        n_permutations = 5000,
        # Let's start with 1
        kernel_width   = 1
    )

explanation

In my case the R-squared value (model_r2) is a little bit low. This is what you want to look at for lime. This is how you investigate your lime models. You want those values as high as possible and you can adjust that using your kernel_width (0.5 or 1.5 gave me better results).

Let’s select the columns, that are important to us.:

explanation %>%
    as.tibble() %>%
    select(feature:prediction)

Because we have selected 8 features, the data lists the top 8 features. feature_value are the values, that were actually used when it ran the model (for example: Overtime is 2 and 2 would be yes. NumCompaniesWorked is 8 and 8 is the numeric value that the employee has worked and so on … ). feature_weight: Magnitude indicates importance. + / - indicates support or contradict. feature_desc is basically just a readable version of the feature importance. The last two columns contain the data that was used.

Visualizing feature importance for a single explanation

Let’s visuale it using plot_features():

g <- plot_features(explanation = explanation, ncol = 1)

Info: 4 < NumCompaniesWorked Note that this label is the result of the continuous variable binning strategy. One of the cuts is at 4, which is how we get this label.

Visualizing Feature Importance For Multiple Explanations

Now we want to scale it up, because often times you are going to be looking at more than one feature that you want to try to explain. If you can do a single explanation, multiple explanations are not much more difficult. Just copy the code frome above and change the slice value:

# 3.3 Multiple Explanations ----

explanation <- test_tbl %>%
    slice(1:20) %>%
    select(-Attrition) %>%
    lime::explain(
        explainer = explainer,
        n_labels   = 1,
        n_features = 8,
        n_permutations = 5000,
        kernel_width   = 0.5
    )

explanation %>%
    as.tibble()

plot_features(explanation, ncol = 4)

The plot will be pretty messy. You can change that a little bit by expanding it. But it will still be a little bit messy and tough to read. It’s too much information to be reported for 20 different cases. If we did only 3 or 4 cases, we could analyze it that way. But for more cases we need a better method to analyze it. That’s why the next function comes into play plot_explanations():

plot_explanations(explanation)

Can you read this plot?

Challenge

This is a two part challenge:

Part 1: Recreate plot_features(). Take the explanation data and use the first case to create a plot similar to the output of plot_features().

explanation %>% 
  as.tibble()
  
case_1 <- explanation %>%
    filter(case == 1)

case_1 %>%
    plot_features()

You will need at least the layers geom_col() and coord_flip().

Bonus Objectives:

Get your custom plot_features() function to scale to multiple cases
Use theme arguments to modify the look of the plot

Part 2: Recreate plot_explanations():

Take the full explanation data and recreate the second plot.

You will need at least the layers geom_tile() and facet_wrap().

HINTS:

If you do get stuck on this challenge, because this is actually a rather difficult challenge, I highly recommend checking out the library lime from Thomas Pedersens’ github page https://github.com/thomasp85/lime. All of the R code is in the folder R. Check that out if you got stuck. You will be able to find some hints in there as to how Thomas did it when he coded the lime package. When coding in the wild, your best friend is GitHub. Use other people’s code as an advantage. Learn from what they do.

Last updated on Mar 16, 2021