Feature Visualization

View colab tutorial

One of the specificities of neural networks is their differentiability. This characteristic allows us to compute gradients, either the gradient of a loss with respect to the parameters, or in the case we are interested in here, of a part of the network with respect to the input. This gradient then allows us to iteratively modify the input in order to maximize an objective such as a neuron, a channel or a combination of objectives.

Quote

If we want to understand individual features, we can search for examples where they have high values either for a neuron at an individual position, or for an entire channel. -- Feature Visualization -- How neural networks build up their understanding of images (2017)¹

More precisely, the explanation of a neuron \(n\) denoted as \(\phi^{(n)}\) is an input \(x* \in \mathcal{X}\) such that

\[ \phi^{(n)} = \underset{x}{arg\ max}\ f(x)^{(n)} - \mathcal{R}(x) \]

with \(f(x)^{(n)}\) the neuron score for an input \(x\) and \(\mathcal{R}(x)\) a regularization term. In practice it turns out that preconditioning the input in a decorrelated space such as the frequency domain allows to obtain more consistent results and to better formulate the regularization (e.g. by controlling the rate of high frequency and low frequency desired).

Examples

Optimize the ten logits of a neural network (we recommend to remove the softmax activation of your network).

from xplique.features_visualizations import Objective
from xplique.features_visualizations import optimize

# load a model...

# targeting the 10 logits of the layer 'logits'
# we can also target a layer by its index, like -1 for the last layer
logits_obj = Objective.neuron(model, "logits", list(range(10)))
images, obj_names = optimize(logits_obj) # 10 images, one for each logits

Create a combination of multiple objectives and aggregate them

from xplique.features_visualizations import Objective
from xplique.features_visualizations import optimize

# load a model...

# target the first logits neuron
logits_obj = Objective.neuron(model, "logits", 0)
# target the third layer
layer_obj = Objective.layer(model, "conv2d_1")
# target the second channel of another layer
channel_obj = Objective.channel(model, "mixed4_2", 2)

# combine the objective
obj = logits_obj * 1.0 + layer_obj * 3.0 + channel_obj * (-5.0)
images, obj_names = optimize(logits_obj) # 1 resulting image

Objective

Use to combine several sub-objectives into one.

__init__(self,
         model: keras.src.engine.training.Model,
         layers: List[keras.src.engine.base_layer.Layer],
         masks: List[tensorflow.python.framework.tensor.Tensor],
         funcs: List[Callable],
         multipliers: List[float],
         names: List[str])

Parameters

• model : keras.src.engine.training.Model

  • Model used for optimization.

• layers : List[keras.src.engine.base_layer.Layer]

  • A list of the layers output for each sub-objectives.

• masks : List[tensorflow.python.framework.tensor.Tensor]

  • A list of masks that will be applied on the targeted layer for each sub-objectives.

• funcs : List[Callable]

  • A list of loss functions for each sub-objectives.

• multipliers : List[float]

  • A list of multiplication factor for each sub-objectives

• names : List[str]

  • A list of name for each sub-objectives

channel(model: keras.src.engine.training.Model,
        layer: Union[str, int],
        channel_ids: Union[int, List[int]],
        multiplier: float = 1.0,
        names: Union[str, List[str], None] = None)

Util to build an objective to maximise a channel.

Parameters

• model : keras.src.engine.training.Model

  • Model used for optimization.

• layer : Union[str, int]

  • Index or name of the targeted layer.

• channel_ids : Union[int, List[int]]

  • Indexes of the channels to maximise.

• multiplier : float = 1.0

  • Multiplication factor of the objectives.

• names : Union[str, List[str], None] = None

  • Names for each objectives.

Return

• objective

  • An objective containing a sub-objective for each channels.

compile(self) -> Tuple[keras.src.engine.training.Model,
        Callable,
        List[str], Tuple]

Compile all the sub-objectives into one and return the objects for the optimisation process.

Return

• model_reconfigured : Tuple[keras.src.engine.training.Model, Callable, List[str], Tuple]

  • Model with the outputs needed for the optimization.

• objective_function : Tuple[keras.src.engine.training.Model, Callable, List[str], Tuple]

  • Function to call that compute the loss for the objectives.

• names : Tuple[keras.src.engine.training.Model, Callable, List[str], Tuple]

  • Names of each objectives.

• input_shape : Tuple[keras.src.engine.training.Model, Callable, List[str], Tuple]

  • Shape of the input, one sample for each optimization.

direction(model: keras.src.engine.training.Model,
          layer: Union[str, int],
          vectors: Union[tensorflow.python.framework.tensor.Tensor, List[tensorflow.python.framework.tensor.Tensor]],
          multiplier: float = 1.0,
          cossim_pow: float = 2.0,
          names: Union[str, List[str], None] = None)

Util to build an objective to maximise a direction of a layer.

Parameters

• model : keras.src.engine.training.Model

  • Model used for optimization.

• layer : Union[str, int]

  • Index or name of the targeted layer.

• vectors : Union[tensorflow.python.framework.tensor.Tensor, List[tensorflow.python.framework.tensor.Tensor]]

  • Direction(s) to optimize.

• multiplier : float = 1.0

  • Multiplication factor of the objective.

• cossim_pow : float = 2.0

  • Power of the cosine similarity, higher value encourage the objective to care more about the angle of the activations.

• names : Union[str, List[str], None] = None

  • A name for each objectives.

Return

• objective

  • An objective ready to be compiled

layer(model: keras.src.engine.training.Model,
      layer: Union[str, int],
      reducer: str = 'magnitude',
      multiplier: float = 1.0,
      name: Optional[str] = None)

Util to build an objective to maximise a layer.

Parameters

• model : keras.src.engine.training.Model

  • Model used for optimization.

• layer : Union[str, int]

  • Index or name of the targeted layer.

• reducer : str = 'magnitude'

  • Type of reduction to apply, 'mean' will optimize the mean value of the layer, 'magnitude' will optimize the mean of the absolute values.

• multiplier : float = 1.0

  • Multiplication factor of the objective.

• name : Optional[str] = None

  • A name for the objective.

Return

• objective

  • An objective ready to be compiled

neuron(model: keras.src.engine.training.Model,
       layer: Union[str, int],
       neurons_ids: Union[int, List[int]],
       multiplier: float = 1.0,
       names: Union[str, List[str], None] = None)

Util to build an objective to maximise a neuron.

Parameters

• model : keras.src.engine.training.Model

  • Model used for optimization.

• layer : Union[str, int]

  • Index or name of the targeted layer.

• neurons_ids : Union[int, List[int]]

  • Indexes of the neurons to maximise.

• multiplier : float = 1.0

  • Multiplication factor of the objectives.

• names : Union[str, List[str], None] = None

  • Names for each objectives.

Return

• objective

  • An objective containing a sub-objective for each neurons.

---

1. Feature Visualization -- How neural networks build up their understanding of images (2017) ↩