Perform sensitivity analysis and model explanations

What is sensitivity analysis?

Sensitivity analysis is the study of how the uncertainty in the output of a model can be divided and allocated to different sources of uncertainty in its inputs. For more information, read the article.

Why is explainability important?

Machine learning is excellent in prediction accuracy, process efficiency, and research productivity. However, computers usually don't explain their predictions. This becomes a barrier to the adoption of machine learning models. If users can't trust a model or a prediction, they will not work with it. Therefore, the issue is how to help users to trust a model.

What are common model explanation types?

Three of the most common model explanation types are as follows:

• Example-based. Within the context of individual predictions, it is natural to ask which points in a training set resemble the test point most closely or influence the prediction.

  The nearest neighbors and methods based on the influence function are archetypal methods leading to example-based explanations.

• Local. Alternatively, we can try to understand an individual prediction by asking if the input has changed slightly and how this affects the model’s prediction.

  Local explanations are typically derived in either of the following ways:

  • From a model directly (for instance, it is valid for sparse linear models)
  • From a local model that approximates the predictive model well in a neighborhood around a specific point

• Global. To understand a model’s overall behavior, it can be helpful to know about the patterns underlying the model’s behavior.

  Global explanations usually take the form of a series of rules.

What are the differences between model explanation types?

Example-based explanations are distinct from the other two types. The former relies on sample data points, whereas the latter two—on features.

Local and global explanations capture fundamentally different characteristics of the predictive model.

Generally, local explanations are better suited for modeling smooth continuous effects. For discontinuous effects or effects that are strong in a small region that can be approximated well by discontinuities, the explanations either fail to detect the effect or make unusual predictions, depending on how the local neighborhood is defined. These effects are global patterns because they are difficult to detect or to model with local explanations.

Conversely, global explanations are better suited for global patterns because these discontinuities create natural rules. They are less effective at explaining continuous effects because explanation rules must introduce arbitrary feature discretization or binning.

Most real datasets have both continuous and discontinuous effects. Therefore, it is crucial to devise explanation systems that can capture, or are at least aware of, both types of effects.