Important Note On Feature Extractors

When using Oodeel, you have to keep in mind how the detector object works, and specifically, how it extracts features from the model that is given as an argument to the .fit() method of OOD baselines inheriting from OODBaseDetector. Under the hood, OODBaseDetector uses an object called FeatureExtractor (with two child versions, KerasFeatureExtractoror TorchFeatureExtractor, depending on your model's implementation).

The key point here is to be able to correctly identify the output layer(s) of your model so that the FeatureExtractor knows what to extract. The layer can be identified by a name or by a slice, if possible. Let's dive into different situations

Important

In this notebook, we go through FeatureExtractor class, but this class is never explicitly used by the user. It works under the hood of OODBaseDetector. Still, understanding how it works is mandatory for correct usage of OODBaseDetector, see the Wrap-up section below.

Tensorflow models¶

In [1]:

import os

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
import tensorflow as tf

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
from oodeel.extractor import KerasFeatureExtractor

from IPython.display import clear_output

import os os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2" import tensorflow as tf tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR) from oodeel.extractor import KerasFeatureExtractor from IPython.display import clear_output

A keras Sequential model¶

In [2]:

def generate_model(input_shape=(32, 32, 3), output_shape=10):
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(4, kernel_size=(2, 2), activation="relu"))
    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
    model.add(tf.keras.layers.Dropout(0.25))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(output_shape))
    model.add(tf.keras.layers.Activation("softmax"))
    model.compile(loss="categorical_crossentropy", optimizer="sgd")

    return model


model = generate_model()
model.compile(optimizer="adam")

clear_output()

def generate_model(input_shape=(32, 32, 3), output_shape=10): model = tf.keras.models.Sequential() model.add(tf.keras.layers.Input(shape=input_shape)) model.add(tf.keras.layers.Conv2D(4, kernel_size=(2, 2), activation="relu")) model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2))) model.add(tf.keras.layers.Dropout(0.25)) model.add(tf.keras.layers.Flatten()) model.add(tf.keras.layers.Dense(output_shape)) model.add(tf.keras.layers.Activation("softmax")) model.compile(loss="categorical_crossentropy", optimizer="sgd") return model model = generate_model() model.compile(optimizer="adam") clear_output()

Let's see what's in there

In [3]:

model.summary()

model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 31, 31, 4)         52        
                                                                 
 max_pooling2d (MaxPooling2D  (None, 15, 15, 4)        0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 15, 15, 4)         0         
                                                                 
 flatten (Flatten)           (None, 900)               0         
                                                                 
 dense (Dense)               (None, 10)                9010      
                                                                 
 activation (Activation)     (None, 10)                0         
                                                                 
=================================================================
Total params: 9,062
Trainable params: 9,062
Non-trainable params: 0
_________________________________________________________________

Most of the time, it is of interest to take the output of neural networks' penultimate layers to apply OOD methods. Here, we can see that the layer can be identified as the $d-3$-th, with $d$ the depth of the network. To achieve that instantiate the FeatureExtractor as:

In [4]:

extractor = KerasFeatureExtractor(model, feature_layers_id=[-3])

x = tf.ones((100, 32, 32, 3))
x_latent, _ = extractor(x)
print(x_latent.shape)

extractor = KerasFeatureExtractor(model, feature_layers_id=[-3]) x = tf.ones((100, 32, 32, 3)) x_latent, _ = extractor(x) print(x_latent.shape)

(100, 900)

Tip

For convenience, the logits are always returned when calling a FeatureExtractor. We do not use them in this notebook, if you want to get them, just replace x_latent, _ = extractor(x) with x_latent, logits = extractor(x)

Alternatively, you can identify the layer by its name:

In [5]:

extractor = KerasFeatureExtractor(model, feature_layers_id=["flatten"])

x = tf.ones((100, 32, 32, 3))
x_latent, _ = extractor(x)
print(x_latent.shape)

extractor = KerasFeatureExtractor(model, feature_layers_id=["flatten"]) x = tf.ones((100, 32, 32, 3)) x_latent, _ = extractor(x) print(x_latent.shape)

(100, 900)

You can also set the starting point of your extractor, which can be useful to avoid repeated forward passes:

In [6]:

extractor_2 = KerasFeatureExtractor(
    model, input_layer_id="dense", feature_layers_id=["activation"]
)

x_latent, _ = extractor(x)
print(x_latent.shape)
y, _ = extractor_2(x_latent)
print(y.shape)

extractor_2 = KerasFeatureExtractor( model, input_layer_id="dense", feature_layers_id=["activation"] ) x_latent, _ = extractor(x) print(x_latent.shape) y, _ = extractor_2(x_latent) print(y.shape)

(100, 900)
(100, 10)

Warning:

• Be careful, the name of the input layer is that of the layer following the previous output layer
• The extractor may only have one input layer (hence the str format of the argument instead of list)

If needed, you can get the output of several layers at the same time:

In [3]:

extractor = KerasFeatureExtractor(model, feature_layers_id=[-3, -1])

x = tf.ones((100, 32, 32, 3))
x_latent, _ = extractor(x)
print(x_latent[0].shape, x_latent[1].shape)

extractor = KerasFeatureExtractor(model, feature_layers_id=[-3, -1]) x = tf.ones((100, 32, 32, 3)) x_latent, _ = extractor(x) print(x_latent[0].shape, x_latent[1].shape)

(100, 900) (100, 10)

Warning:

For this cell to work, you have to clear ipython kernel from the previous extractors

For non-sequential keras models¶

When your model is built out of a set of layers that are not connected sequentially, your only option is to rely on the identification by layer name, as referred in model.summary()

PyTorch Models¶

In [2]:

import torch
import torch.nn as nn
import torch.nn.functional as F

from oodeel.extractor import TorchFeatureExtractor

import torch import torch.nn as nn import torch.nn.functional as F from oodeel.extractor import TorchFeatureExtractor

Now let's consider some Pytorch model

In [3]:

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1)  # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


model = Net()

class Net(nn.Module): def __init__(self): super().__init__() self.conv1 = nn.Conv2d(3, 6, 5) self.pool = nn.MaxPool2d(2, 2) self.conv2 = nn.Conv2d(6, 16, 5) self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(120, 84) self.fc3 = nn.Linear(84, 10) def forward(self, x): x = self.pool(F.relu(self.conv1(x))) x = self.pool(F.relu(self.conv2(x))) x = torch.flatten(x, 1) # flatten all dimensions except batch x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) x = self.fc3(x) return x model = Net()

Similarly, let's display how the model is constructed:

In [4]:

for layer in model.named_modules():
    print(layer)

for layer in model.named_modules(): print(layer)

('', Net(
  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
))
('conv1', Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1)))
('pool', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
('conv2', Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1)))
('fc1', Linear(in_features=400, out_features=120, bias=True))
('fc2', Linear(in_features=120, out_features=84, bias=True))
('fc3', Linear(in_features=84, out_features=10, bias=True))

That case is pretty much the same as for Tensorflow:

In [5]:

extractor = TorchFeatureExtractor(model, feature_layers_id=[-3])

x = torch.ones((100, 3, 32, 32))
x_latent, _ = extractor(x)
print("numbered output:\n", x_latent.shape)

extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"])

x = torch.ones((100, 3, 32, 32))
x_latent, _ = extractor(x)
print("named output:\n", x_latent.shape)

extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1", "fc3"])

x = torch.ones((100, 3, 32, 32))
x_latent, _ = extractor(x)
print("multi output:\n", x_latent[0].shape, x_latent[1].shape)

extractor = TorchFeatureExtractor(model, feature_layers_id=[-3]) x = torch.ones((100, 3, 32, 32)) x_latent, _ = extractor(x) print("numbered output:\n", x_latent.shape) extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"]) x = torch.ones((100, 3, 32, 32)) x_latent, _ = extractor(x) print("named output:\n", x_latent.shape) extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1", "fc3"]) x = torch.ones((100, 3, 32, 32)) x_latent, _ = extractor(x) print("multi output:\n", x_latent[0].shape, x_latent[1].shape)

numbered output:
 torch.Size([100, 120])
named output:
 torch.Size([100, 120])
multi output:
 torch.Size([100, 120]) torch.Size([100, 10])

Warning:

As opposed to Tensorflow, PyTorch extractor can only take internal layers as input for nn.Sequential models.

Will not work:

In [ ]:

extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"])
extractor_2 = TorchFeatureExtractor(
    model, input_layer_id="fc2", feature_layers_id=["fc3"]
)

x_latent, _ = extractor(x)
print(x_latent.shape)
y, _ = extractor_2(x_latent)
print(y.shape)

extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"]) extractor_2 = TorchFeatureExtractor( model, input_layer_id="fc2", feature_layers_id=["fc3"] ) x_latent, _ = extractor(x) print(x_latent.shape) y, _ = extractor_2(x_latent) print(y.shape)

Will work:

In [8]:

from collections import OrderedDict


def named_sequential_model():
    return nn.Sequential(
        OrderedDict(
            [
                ("conv1", nn.Conv2d(3, 6, 5)),
                ("relu1", nn.ReLU()),
                ("pool1", nn.MaxPool2d(2, 2)),
                ("conv2", nn.Conv2d(6, 16, 5)),
                ("relu2", nn.ReLU()),
                ("pool2", nn.MaxPool2d(2, 2)),
                ("flatten", nn.Flatten()),
                ("fc1", nn.Linear(16 * 5 * 5, 120)),
                ("fc2", nn.Linear(120, 84)),
                ("fc3", nn.Linear(84, 10)),
            ]
        )
    )


model = named_sequential_model()
extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"])

x = torch.ones((100, 3, 32, 32))
x_latent, _ = extractor(x)
print(x_latent.shape)

# Be careful, once extractor_2 is instanciated, extractor no longer works
# because hooks are cleaned
extractor_2 = TorchFeatureExtractor(
    model, input_layer_id="fc2", feature_layers_id=["fc3"]
)
y, _ = extractor_2(x_latent)
print(y.shape)

from collections import OrderedDict def named_sequential_model(): return nn.Sequential( OrderedDict( [ ("conv1", nn.Conv2d(3, 6, 5)), ("relu1", nn.ReLU()), ("pool1", nn.MaxPool2d(2, 2)), ("conv2", nn.Conv2d(6, 16, 5)), ("relu2", nn.ReLU()), ("pool2", nn.MaxPool2d(2, 2)), ("flatten", nn.Flatten()), ("fc1", nn.Linear(16 * 5 * 5, 120)), ("fc2", nn.Linear(120, 84)), ("fc3", nn.Linear(84, 10)), ] ) ) model = named_sequential_model() extractor = TorchFeatureExtractor(model, feature_layers_id=["fc1"]) x = torch.ones((100, 3, 32, 32)) x_latent, _ = extractor(x) print(x_latent.shape) # Be careful, once extractor_2 is instanciated, extractor no longer works # because hooks are cleaned extractor_2 = TorchFeatureExtractor( model, input_layer_id="fc2", feature_layers_id=["fc3"] ) y, _ = extractor_2(x_latent) print(y.shape)

torch.Size([100, 120])
torch.Size([100, 10])

Wrap-up¶

Once you have identified the way you want to extract data from your model, which boils down to correctly identifying what to put under the feature_layers_id and input_layer_id arguments, you can properly instantiate your OODBaseDetector like this (example with DKNN):

In [ ]:

from oodeel.methods import DKNN

dknn = DKNN()
dknn.fit(
    model, 
    feature_layers_id=["fc1"], 
    #some_fit_dataset)
)

from oodeel.methods import DKNN dknn = DKNN() dknn.fit( model, feature_layers_id=["fc1"], #some_fit_dataset) )

In fact, you never have to instantiate the feature extractor by yourself, it is automatically performed by OODBaseDetector (here DKNN). It detects the underlying library of model, and instantiates the adapted FeatureExtractor using the arguments given as input to OODBaseDetector.

Note:

We recommend that you only identify layers by name; we found it to be less error prone.