Plots

plot_2D_features(model, in_dataset, output_layer_id, out_dataset=None, proj_method='TSNE', max_samples=4000, title=None, **proj_kwargs)

Visualize ID and OOD features of a model on a 2D plan using dimensionality reduction methods and matplotlib scatter function. Different projection methods are available: TSNE, PCA.

Parameters:

Name	Type	Description	Default
model	Callable	Torch or Keras model.	required
in_dataset	DatasetType	In-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space.	required
output_layer_id	Union[int, str]	Identifier for the layer to inspect.	required
out_dataset	DatasetType	Out-of-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space if not equal to None. Defaults to None.	None
proj_method	str	Projection method for 2d dimensionality reduction. Defaults to "TSNE", alternative: "PCA".	'TSNE'
max_samples	int	Max samples to display on the scatter plot. Defaults to 4000.	4000
title	str	Custom figure title. Defaults to None.	None

Source code in oodeel/eval/plots/features.py

def plot_2D_features(
    model: Callable,
    in_dataset: DatasetType,
    output_layer_id: Union[int, str],
    out_dataset: DatasetType = None,
    proj_method: str = "TSNE",
    max_samples: int = 4000,
    title: str = None,
    **proj_kwargs,
):
    """Visualize ID and OOD features of a model on a 2D plan using dimensionality
    reduction methods and matplotlib scatter function. Different projection methods are
    available: TSNE, PCA.

    Args:
        model (Callable): Torch or Keras model.
        in_dataset (DatasetType): In-distribution dataset (torch dataloader or tf
            dataset) that will be projected on the model feature space.
        output_layer_id (Union[int, str]): Identifier for the layer to inspect.
        out_dataset (DatasetType, optional): Out-of-distribution dataset (torch
            dataloader or tf dataset) that will be projected on the model feature space
            if not equal to None. Defaults to None.
        proj_method (str, optional): Projection method for 2d dimensionality reduction.
            Defaults to "TSNE", alternative: "PCA".
        max_samples (int, optional): Max samples to display on the scatter plot.
            Defaults to 4000.
        title (str, optional): Custom figure title. Defaults to None.
    """

    _plot_features(
        model=model,
        in_dataset=in_dataset,
        output_layer_id=output_layer_id,
        out_dataset=out_dataset,
        proj_method=proj_method,
        max_samples=max_samples,
        title=title,
        n_components=2,
        **proj_kwargs,
    )

plot_3D_features(model, in_dataset, output_layer_id, out_dataset=None, proj_method='TSNE', max_samples=4000, title=None, **proj_kwargs)

Visualize ID and OOD features of a model on a 3D space using dimensionality reduction methods and matplotlib scatter function. Different projection methods are available: TSNE, PCA.

Parameters:

Name	Type	Description	Default
model	Callable	Torch or Keras model.	required
in_dataset	DatasetType	In-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space.	required
output_layer_id	Union[int, str]	Identifier for the layer to inspect.	required
out_dataset	DatasetType	Out-of-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space if not equal to None. Defaults to None.	None
proj_method	str	Projection method for 2d dimensionality reduction. Defaults to "TSNE", alternative: "PCA".	'TSNE'
max_samples	int	Max samples to display on the scatter plot. Defaults to 4000.	4000
title	str	Custom figure title. Defaults to None.	None

Source code in oodeel/eval/plots/features.py

def plot_3D_features(
    model: Callable,
    in_dataset: DatasetType,
    output_layer_id: Union[int, str],
    out_dataset: DatasetType = None,
    proj_method: str = "TSNE",
    max_samples: int = 4000,
    title: str = None,
    **proj_kwargs,
):
    """Visualize ID and OOD features of a model on a 3D space using dimensionality
    reduction methods and matplotlib scatter function. Different projection methods are
    available: TSNE, PCA.

    Args:
        model (Callable): Torch or Keras model.
        in_dataset (DatasetType): In-distribution dataset (torch dataloader or tf
            dataset) that will be projected on the model feature space.
        output_layer_id (Union[int, str]): Identifier for the layer to inspect.
        out_dataset (DatasetType, optional): Out-of-distribution dataset (torch
            dataloader or tf dataset) that will be projected on the model feature space
            if not equal to None. Defaults to None.
        proj_method (str, optional): Projection method for 2d dimensionality reduction.
            Defaults to "TSNE", alternative: "PCA".
        max_samples (int, optional): Max samples to display on the scatter plot.
            Defaults to 4000.
        title (str, optional): Custom figure title. Defaults to None.
    """
    _plot_features(
        model=model,
        in_dataset=in_dataset,
        output_layer_id=output_layer_id,
        out_dataset=out_dataset,
        proj_method=proj_method,
        max_samples=max_samples,
        title=title,
        n_components=3,
        **proj_kwargs,
    )

plot_ood_scores(scores_in, scores_out, log_scale=False, title=None)

Plot histograms of OOD detection scores for ID and OOD distribution, using matplotlib and seaborn.

Parameters:

Name	Type	Description	Default
scores_in	ndarray	OOD detection scores for ID data.	required
scores_out	ndarray	OOD detection scores for OOD data.	required
log_scale	bool	If True, apply a log scale on x axis. Defaults to False.	False
title	str	Custom figure title. If None a default one is provided. Defaults to None.	None

Source code in oodeel/eval/plots/metrics.py

def plot_ood_scores(
    scores_in: np.ndarray,
    scores_out: np.ndarray,
    log_scale: bool = False,
    title: str = None,
):
    """Plot histograms of OOD detection scores for ID and OOD distribution, using
    matplotlib and seaborn.

    Args:
        scores_in (np.ndarray): OOD detection scores for ID data.
        scores_out (np.ndarray): OOD detection scores for OOD data.
        log_scale (bool, optional): If True, apply a log scale on x axis. Defaults to
            False.
        title (str, optional): Custom figure title. If None a default one is provided.
            Defaults to None.
    """
    title = title or "Histograms of OOD detection scores"
    ax1 = sns.histplot(
        data=scores_in,
        alpha=0.5,
        label="ID data",
        stat="density",
        log_scale=log_scale,
        kde=True,
    )
    ax2 = sns.histplot(
        data=scores_out,
        alpha=0.5,
        label="OOD data",
        stat="density",
        log_scale=log_scale,
        kde=True,
    )
    ymax = max(ax1.get_ylim()[1], ax2.get_ylim()[1])
    threshold = np.percentile(scores_out, q=5.0)
    plt.vlines(
        x=[threshold],
        ymin=0,
        ymax=ymax,
        colors=["red"],
        linestyles=["dashed"],
        alpha=0.7,
        label="TPR=95%",
    )
    plt.xlabel("OOD score")
    plt.legend()
    plt.title(title, weight="bold").set_fontsize(11)

plot_roc_curve(scores_in, scores_out, title=None)

Plot ROC curve for OOD detection task, using matplotlib and seaborn.

Parameters:

Name	Type	Description	Default
scores_in	ndarray	OOD detection scores for ID data.	required
scores_out	ndarray	OOD detection scores for OOD data.	required
title	str	Custom figure title. If None a default one is provided. Defaults to None.	None

Source code in oodeel/eval/plots/metrics.py

def plot_roc_curve(scores_in: np.ndarray, scores_out: np.ndarray, title: str = None):
    """Plot ROC curve for OOD detection task, using matplotlib and seaborn.

    Args:
        scores_in (np.ndarray): OOD detection scores for ID data.
        scores_out (np.ndarray): OOD detection scores for OOD data.
        title (str, optional): Custom figure title. If None a default one is provided.
            Defaults to None.
    """
    # compute auroc
    metrics = bench_metrics(
        (scores_in, scores_out),
        metrics=["auroc", "fpr95tpr"],
    )
    auroc, fpr95tpr = metrics["auroc"], metrics["fpr95tpr"]

    # roc
    fpr, tpr, _, _, _ = get_curve(
        scores=np.concatenate([scores_in, scores_out]),
        labels=np.concatenate([scores_in * 0 + 0, scores_out * 0 + 1]),
    )

    # plot roc
    title = title or "ROC curve (AuC = {:.3f})".format(auroc)
    plt.plot(fpr, tpr)
    plt.fill_between(fpr, tpr, np.zeros_like(tpr), alpha=0.5)
    plt.plot([fpr95tpr, fpr95tpr, 0], [0, 0.95, 0.95], "--", color="red", alpha=0.7)
    plt.scatter([fpr95tpr], [0.95], marker="o", alpha=0.7, color="red", label="TPR=95%")
    plt.xlabel("FPR")
    plt.ylabel("TPR")
    plt.xlim([-0.01, 1.01])
    plt.ylim([-0.01, 1.01])
    plt.legend()
    plt.title(title, weight="bold").set_fontsize(11)

plotly_3D_features(model, in_dataset, output_layer_id, out_dataset=None, proj_method='TSNE', max_samples=4000, title=None, **proj_kwargs)

Visualize ID and OOD features of a model on a 3D space using dimensionality reduction methods and matplotlib scatter function. Different projection methods are available: TSNE, PCA. This function requires the package plotly to be installed to run an interactive 3D scatter plot.

Parameters:

Name	Type	Description	Default
model	Callable	Torch or Keras model.	required
in_dataset	DatasetType	In-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space.	required
output_layer_id	Union[int, str]	Identifier for the layer to inspect.	required
out_dataset	DatasetType	Out-of-distribution dataset (torch dataloader or tf dataset) that will be projected on the model feature space if not equal to None. Defaults to None.	None
proj_method	str	Projection method for 2d dimensionality reduction. Defaults to "TSNE", alternative: "PCA".	'TSNE'
max_samples	int	Max samples to display on the scatter plot. Defaults to 4000.	4000
title	str	Custom figure title. Defaults to None.	None

Source code in oodeel/eval/plots/plotly.py

def plotly_3D_features(
    model: Callable,
    in_dataset: DatasetType,
    output_layer_id: Union[int, str],
    out_dataset: DatasetType = None,
    proj_method: str = "TSNE",
    max_samples: int = 4000,
    title: str = None,
    **proj_kwargs,
):
    """Visualize ID and OOD features of a model on a 3D space using dimensionality
    reduction methods and matplotlib scatter function. Different projection methods are
    available: TSNE, PCA. This function requires the package plotly to be installed to
    run an interactive 3D scatter plot.

    Args:
        model (Callable): Torch or Keras model.
        in_dataset (DatasetType): In-distribution dataset (torch dataloader or tf
            dataset) that will be projected on the model feature space.
        output_layer_id (Union[int, str]): Identifier for the layer to inspect.
        out_dataset (DatasetType, optional): Out-of-distribution dataset (torch
            dataloader or tf dataset) that will be projected on the model feature space
            if not equal to None. Defaults to None.
        proj_method (str, optional): Projection method for 2d dimensionality reduction.
            Defaults to "TSNE", alternative: "PCA".
        max_samples (int, optional): Max samples to display on the scatter plot.
            Defaults to 4000.
        title (str, optional): Custom figure title. Defaults to None.
    """
    max_samples = max_samples if out_dataset is None else max_samples // 2

    # feature extractor
    _, _, op, FeatureExtractorClass = import_backend_specific_stuff(model)
    feature_extractor = FeatureExtractorClass(model, [output_layer_id])

    # === extract id features ===
    # features
    in_features, _ = feature_extractor.predict(in_dataset)
    in_features = op.convert_to_numpy(op.flatten(in_features[0]))[:max_samples]

    # labels
    in_labels = []
    for _, batch_y in in_dataset:
        in_labels.append(op.convert_to_numpy(batch_y))
    in_labels = np.concatenate(in_labels)[:max_samples]
    in_labels = list(map(lambda x: f"class {x}", in_labels))

    # === extract ood features ===
    if out_dataset is not None:
        # features
        out_features, _ = feature_extractor.predict(out_dataset)
        out_features = op.convert_to_numpy(op.flatten(out_features[0]))[:max_samples]

        # labels
        out_labels = np.array(["unknown"] * len(out_features))

        # concatenate id and ood items
        features = np.concatenate([out_features, in_features])
        labels = np.concatenate([out_labels, in_labels])
        data_type = np.array(["OOD"] * len(out_labels) + ["ID"] * len(in_labels))
        points_size = np.array([1] * len(out_labels) + [3] * len(in_labels))
    else:
        features = in_features
        labels = in_labels
        data_type = np.array(["ID"] * len(in_labels))
        points_size = np.array([3] * len(in_labels))

    # === project on 3d space using tsne or pca ===
    proj_class = PROJ_DICT[proj_method]["class"]
    p_kwargs = PROJ_DICT[proj_method]["default_kwargs"]
    p_kwargs.update(proj_kwargs)
    projector = proj_class(
        n_components=3,
        **p_kwargs,
    )
    features_proj = projector.fit_transform(features)

    # === plot 3d features ===
    features_dim = features.shape[1]
    method_str = PROJ_DICT[proj_method]["name"]
    title = (
        title
        or f"{method_str} 3D projection\n"
        + f"[layer {output_layer_id}, dim: {features_dim}]"
    )

    x, y, z = features_proj.T
    df = pd.DataFrame(
        {
            "dim 1": x,
            "dim 2": y,
            "dim 3": z,
            "class": labels,
            "data type": data_type,
            "size": points_size,
        }
    )

    # 3D projection
    fig = px.scatter_3d(
        data_frame=df,
        x="dim 1",
        y="dim 2",
        z="dim 3",
        color="class",
        symbol="data type",
        size="size",
        opacity=1,
        category_orders={"class": np.unique(df["class"])},
        symbol_map={"OOD": "circle", "ID": "diamond"},
    )

    fig.update_layout(
        title={"text": title, "y": 0.9, "x": 0.5, "xanchor": "center", "yanchor": "top"}
    )
    fig.show()