deel.lip.metrics

This module contains metrics applicable in provable robustness. See https://arxiv.org/abs/2006.06520 and https://arxiv.org/abs/2108.04062 for more information.

BinaryProvableAvgRobustness

BinaryProvableAvgRobustness(
    lip_const=1.0,
    negative_robustness=False,
    reduction=Reduction.AUTO,
    name="BinaryProvableAvgRobustness",
)

Bases: Loss

Compute the average provable robustness radius on the dataset.

\[ \mathbb{E}_{x \in D}\left[ \frac{\phi\left(\mathcal{M}_f(x)\right)}{L_f}\right] \]

\(\mathcal{M}_f(x)\) is a term that: is positive when x is correctly classified and negative otherwise. In both case the value give the robustness radius around x.

In the binary classification setup we have:

\[ \mathcal{M}_f(x) = f(x) \text{ if } l=1, -f(x) \text{otherwise} \]

Where \(D\) is the dataset, \(l\) is the correct label for x and \(L_f\) is the lipschitz constant of the network..

When negative_robustness is set to True misclassified elements count as negative robustness (\(\phi\) act as identity function), when set to False, misclassified elements yield a robustness radius of 0 ( \(\phi(x)=relu( x)\) ). The elements are not ignored when computing the mean in both cases.

This metric works for labels both in {1,0} and {1,-1}.

PARAMETER	DESCRIPTION
lip_const	lipschitz constant of the network TYPE: float DEFAULT: 1.0
reduction	the recution method when training in a multi-gpu / TPU system DEFAULT: AUTO
name	metrics name. TYPE: str DEFAULT: 'BinaryProvableAvgRobustness'

Source code in deel/lip/metrics.py

def __init__(
    self,
    lip_const=1.0,
    negative_robustness=False,
    reduction=Reduction.AUTO,
    name="BinaryProvableAvgRobustness",
):
    r"""

    Compute the average provable robustness radius on the dataset.

    $$
    \mathbb{E}_{x \in D}\left[ \frac{\phi\left(\mathcal{M}_f(x)\right)}{L_f}\right]
    $$

    $\mathcal{M}_f(x)$ is a term that: is positive when x is correctly
    classified and negative otherwise. In both case the value give the robustness
    radius around x.

    In the binary classification setup we have:

    $$
    \mathcal{M}_f(x) = f(x) \text{ if } l=1, -f(x) \text{otherwise}
    $$

    Where $D$ is the dataset, $l$ is the correct label for x and
    $L_f$ is the lipschitz constant of the network..

    When `negative_robustness` is set to `True` misclassified elements count as
    negative robustness ($\phi$ act as identity function), when set to
    `False`,
    misclassified elements yield a robustness radius of 0 ( $\phi(x)=relu(
    x)$ ). The elements are not ignored when computing the mean in both cases.

    This metric works for labels both in {1,0} and {1,-1}.

    Args:
        lip_const (float): lipschitz constant of the network
        reduction: the recution method when training in a multi-gpu / TPU system
        name (str): metrics name.
    """
    self.lip_const = lip_const
    self.negative_robustness = negative_robustness
    if self.negative_robustness:
        self.delta_correction = lambda delta: delta
    else:
        self.delta_correction = tf.nn.relu
    super(BinaryProvableAvgRobustness, self).__init__(reduction, name)

BinaryProvableRobustAccuracy

BinaryProvableRobustAccuracy(
    epsilon=36 / 255,
    lip_const=1.0,
    reduction=Reduction.AUTO,
    name="BinaryProvableRobustAccuracy",
)

Bases: Loss

The accuracy that can be proved at a given epsilon.

PARAMETER	DESCRIPTION
epsilon	the metric will return the guaranteed accuracy for the radius epsilon. TYPE: float DEFAULT: 36 / 255
lip_const	lipschitz constant of the network TYPE: float DEFAULT: 1.0
reduction	the recution method when training in a multi-gpu / TPU system DEFAULT: AUTO
name	metrics name. TYPE: str DEFAULT: 'BinaryProvableRobustAccuracy'

Source code in deel/lip/metrics.py

def __init__(
    self,
    epsilon=36 / 255,
    lip_const=1.0,
    reduction=Reduction.AUTO,
    name="BinaryProvableRobustAccuracy",
):
    r"""

    The accuracy that can be proved at a given epsilon.

    Args:
        epsilon (float): the metric will return the guaranteed accuracy for the
            radius epsilon.
        lip_const (float): lipschitz constant of the network
        reduction: the recution method when training in a multi-gpu / TPU system
        name (str): metrics name.
    """
    self.lip_const = lip_const
    self.epsilon = epsilon
    super(BinaryProvableRobustAccuracy, self).__init__(reduction, name)

CategoricalProvableAvgRobustness

CategoricalProvableAvgRobustness(
    lip_const=1.0,
    disjoint_neurons=True,
    negative_robustness=False,
    reduction=Reduction.AUTO,
    name="CategoricalProvableAvgRobustness",
)

Bases: Loss

Compute the average provable robustness radius on the dataset.

\[ \mathbb{E}_{x \in D}\left[ \frac{\phi\left(\mathcal{M}_f(x)\right)}{L_f}\right] \]

\(\mathcal{M}_f(x)\) is a term that: is positive when x is correctly classified and negative otherwise. In both case the value give the robustness radius around x.

In the multiclass setup we have:

\[ \mathcal{M}_f(x) =f_l(x) - \max_{i \neq l} f_i(x) \]

Where \(D\) is the dataset, \(l\) is the correct label for x and \(L_f\) is the lipschitz constant of the network (\(L = 2 \times \text{lip_const}\) when disjoint_neurons=True, \(L = \sqrt{2} \times \text{lip_const}\) otherwise).

When negative_robustness is set to True misclassified elements count as negative robustness (\(\phi\) act as identity function), when set to False, misclassified elements yield a robustness radius of 0 ( \(\phi(x)=relu( x)\) ). The elements are not ignored when computing the mean in both cases.

This metric works for labels both in {1,0} and {1,-1}.

PARAMETER	DESCRIPTION
lip_const	lipschitz constant of the network TYPE: float DEFAULT: 1.0
disjoint_neurons	must be set to True is your model ends with a FrobeniusDense layer with disjoint_neurons set to True. Set to False otherwise TYPE: bool DEFAULT: True
reduction	the recution method when training in a multi-gpu / TPU system DEFAULT: AUTO
name	metrics name. TYPE: str DEFAULT: 'CategoricalProvableAvgRobustness'

Source code in deel/lip/metrics.py

def __init__(
    self,
    lip_const=1.0,
    disjoint_neurons=True,
    negative_robustness=False,
    reduction=Reduction.AUTO,
    name="CategoricalProvableAvgRobustness",
):
    r"""

    Compute the average provable robustness radius on the dataset.

    $$
    \mathbb{E}_{x \in D}\left[ \frac{\phi\left(\mathcal{M}_f(x)\right)}{L_f}\right]
    $$

    $\mathcal{M}_f(x)$ is a term that: is positive when x is correctly
    classified and negative otherwise. In both case the value give the robustness
    radius around x.

    In the multiclass setup we have:

    $$
    \mathcal{M}_f(x) =f_l(x) - \max_{i \neq l} f_i(x)
    $$

    Where $D$ is the dataset, $l$ is the correct label for x and
    $L_f$ is the lipschitz constant of the network ($L = 2 \times
    \text{lip_const}$ when `disjoint_neurons=True`, $L = \sqrt{2} \times
    \text{lip_const}$ otherwise).

    When `negative_robustness` is set to `True` misclassified elements count as
    negative robustness ($\phi$ act as identity function), when set to
    `False`,
    misclassified elements yield a robustness radius of 0 ( $\phi(x)=relu(
    x)$ ). The elements are not ignored when computing the mean in both cases.

    This metric works for labels both in {1,0} and {1,-1}.

    Args:
        lip_const (float): lipschitz constant of the network
        disjoint_neurons (bool): must be set to True is your model ends with a
            FrobeniusDense layer with `disjoint_neurons` set to True. Set to False
            otherwise
        reduction: the recution method when training in a multi-gpu / TPU system
        name (str): metrics name.
    """
    self.lip_const = lip_const
    self.disjoint_neurons = disjoint_neurons
    self.negative_robustness = negative_robustness
    if disjoint_neurons:
        self.certificate_factor = 2 * lip_const
    else:
        self.certificate_factor = math.sqrt(2) * lip_const
    if self.negative_robustness:
        self.delta_correction = lambda delta: delta
    else:
        self.delta_correction = tf.nn.relu
    super(CategoricalProvableAvgRobustness, self).__init__(reduction, name)

CategoricalProvableRobustAccuracy

CategoricalProvableRobustAccuracy(
    epsilon=36 / 255,
    lip_const=1.0,
    disjoint_neurons=True,
    reduction=Reduction.AUTO,
    name="CategoricalProvableRobustAccuracy",
)

Bases: Loss

The accuracy that can be proved at a given epsilon.

PARAMETER	DESCRIPTION
epsilon	the metric will return the guaranteed accuracy for the radius epsilon. TYPE: float DEFAULT: 36 / 255
lip_const	lipschitz constant of the network TYPE: float DEFAULT: 1.0
disjoint_neurons	must be set to True if your model ends with a FrobeniusDense layer with disjoint_neurons set to True. Set to False otherwise TYPE: bool DEFAULT: True
reduction	the recution method when training in a multi-gpu / TPU system DEFAULT: AUTO
name	metrics name. TYPE: str DEFAULT: 'CategoricalProvableRobustAccuracy'

Source code in deel/lip/metrics.py

def __init__(
    self,
    epsilon=36 / 255,
    lip_const=1.0,
    disjoint_neurons=True,
    reduction=Reduction.AUTO,
    name="CategoricalProvableRobustAccuracy",
):
    r"""

    The accuracy that can be proved at a given epsilon.

    Args:
        epsilon (float): the metric will return the guaranteed accuracy for the
            radius epsilon.
        lip_const (float): lipschitz constant of the network
        disjoint_neurons (bool): must be set to True if your model ends with a
            FrobeniusDense layer with `disjoint_neurons` set to True. Set to False
            otherwise
        reduction: the recution method when training in a multi-gpu / TPU system
        name (str): metrics name.
    """
    self.lip_const = lip_const
    self.epsilon = epsilon
    self.disjoint_neurons = disjoint_neurons
    if disjoint_neurons:
        self.certificate_factor = 2 * lip_const
    else:
        self.certificate_factor = math.sqrt(2) * lip_const
    super(CategoricalProvableRobustAccuracy, self).__init__(reduction, name)