deel.lip.losses

This module contains losses used in Wasserstein distance estimation. See this paper for more information.

CategoricalHinge

CategoricalHinge(
    min_margin,
    reduction=Reduction.AUTO,
    name="CategoricalHinge",
)

Bases: Loss

Similar to original categorical hinge, but with a settable margin parameter. This implementation is sligthly different from the Keras one.

y_true and y_pred must be of shape (batch_size, # classes). Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
min_margin	margin parameter. TYPE: float
reduction	reduction of the loss, passed to original loss. DEFAULT: AUTO
name	name of the loss TYPE: str DEFAULT: 'CategoricalHinge'

Source code in deel/lip/losses.py

def __init__(self, min_margin, reduction=Reduction.AUTO, name="CategoricalHinge"):
    """
    Similar to original categorical hinge, but with a settable margin parameter.
    This implementation is sligthly different from the Keras one.

    `y_true` and `y_pred` must be of shape (batch_size, # classes).
    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        min_margin (float): margin parameter.
        reduction: reduction of the loss, passed to original loss.
        name (str): name of the loss
    """
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    super(CategoricalHinge, self).__init__(name=name, reduction=reduction)

HKR

HKR(
    alpha,
    min_margin=1.0,
    multi_gpu=False,
    reduction=Reduction.AUTO,
    name="HKR",
)

Bases: Loss

Wasserstein loss with a regularization parameter based on the hinge margin loss.

\[ \inf_{f \in Lip_1(\Omega)} \underset{\textbf{x} \sim P_-}{\mathbb{E}} \left[f(\textbf{x} )\right] - \underset{\textbf{x} \sim P_+} {\mathbb{E}} \left[f(\textbf{x} )\right] + \alpha \underset{\textbf{x}}{\mathbb{E}} \left(\text{min_margin} -Yf(\textbf{x})\right)_+ \]

Note that y_true and y_pred must be of rank 2: (batch_size, 1) or (batch_size, C) for multilabel classification (with C categories). y_true accepts label values in (0, 1), (-1, 1), or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

Using a multi-GPU/TPU strategy requires to set multi_gpu to True and to pre-process the labels y_true with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
alpha	regularization factor TYPE: float
min_margin	minimal margin ( see hinge_margin_loss ) Kantorovich-Rubinstein term of the loss. In order to be consistent between hinge and KR, the first label must yield the positive class while the second yields negative class. TYPE: float DEFAULT: 1.0
multi_gpu	set to True when running on multi-GPU/TPU TYPE: bool DEFAULT: False
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'HKR'

Source code in deel/lip/losses.py

def __init__(
    self,
    alpha,
    min_margin=1.0,
    multi_gpu=False,
    reduction=Reduction.AUTO,
    name="HKR",
):
    r"""
    Wasserstein loss with a regularization parameter based on the hinge margin loss.

    $$
    \inf_{f \in Lip_1(\Omega)} \underset{\textbf{x} \sim P_-}{\mathbb{E}}
    \left[f(\textbf{x} )\right] - \underset{\textbf{x}  \sim P_+}
    {\mathbb{E}} \left[f(\textbf{x} )\right] + \alpha
    \underset{\textbf{x}}{\mathbb{E}} \left(\text{min_margin}
    -Yf(\textbf{x})\right)_+
    $$

    Note that `y_true` and `y_pred` must be of rank 2: (batch_size, 1) or
    (batch_size, C) for multilabel classification (with C categories).
    `y_true` accepts label values in (0, 1), (-1, 1), or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Using a multi-GPU/TPU strategy requires to set `multi_gpu` to True and to
    pre-process the labels `y_true` with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        alpha (float): regularization factor
        min_margin (float): minimal margin ( see hinge_margin_loss )
            Kantorovich-Rubinstein term of the loss. In order to be consistent
            between hinge and KR, the first label must yield the positive class
            while the second yields negative class.
        multi_gpu (bool): set to True when running on multi-GPU/TPU
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.alpha = tf.Variable(alpha, dtype=tf.float32)
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    self.multi_gpu = multi_gpu
    self.KRloss = KR(multi_gpu=multi_gpu)
    if alpha == np.inf:  # alpha = inf => hinge only
        self.fct = partial(hinge_margin, min_margin=self.min_margin)
    else:
        self.fct = self.hkr
    super(HKR, self).__init__(reduction=reduction, name=name)

HingeMargin

HingeMargin(
    min_margin=1.0,
    reduction=Reduction.AUTO,
    name="HingeMargin",
)

Bases: Loss

Compute the hinge margin loss.

\[ \underset{\textbf{x}}{\mathbb{E}} \left(\text{min_margin} -Yf(\textbf{x})\right)_+ \]

Note that y_true and y_pred must be of rank 2: (batch_size, 1) or (batch_size, C) for multilabel classification (with C categories). y_true accepts label values in (0, 1), (-1, 1), or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
min_margin	margin to enforce. TYPE: float DEFAULT: 1.0
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'HingeMargin'

Source code in deel/lip/losses.py

def __init__(self, min_margin=1.0, reduction=Reduction.AUTO, name="HingeMargin"):
    r"""
    Compute the hinge margin loss.

    $$
    \underset{\textbf{x}}{\mathbb{E}} \left(\text{min_margin}
    -Yf(\textbf{x})\right)_+
    $$

    Note that `y_true` and `y_pred` must be of rank 2: (batch_size, 1) or
    (batch_size, C) for multilabel classification (with C categories).
    `y_true` accepts label values in (0, 1), (-1, 1), or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        min_margin (float): margin to enforce.
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    super(HingeMargin, self).__init__(reduction=reduction, name=name)

KR

KR(multi_gpu=False, reduction=Reduction.AUTO, name='KR')

Bases: Loss

Loss to estimate Wasserstein-1 distance using Kantorovich-Rubinstein duality. The Kantorovich-Rubinstein duality is formulated as following:

\[ W_1(\mu, \nu) = \sup_{f \in Lip_1(\Omega)} \underset{\textbf{x} \sim \mu}{\mathbb{E}} \left[f(\textbf{x} )\right] - \underset{\textbf{x} \sim \nu}{\mathbb{E}} \left[f(\textbf{x} )\right] \]

Where mu and nu stands for the two distributions, the distribution where the label is 1 and the rest.

Note that y_true and y_pred must be of rank 2: (batch_size, 1) or (batch_size, C) for multilabel classification (with C categories). y_true accepts label values in (0, 1), (-1, 1), or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

Using a multi-GPU/TPU strategy requires to set multi_gpu to True and to pre-process the labels y_true with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
multi_gpu	set to True when running on multi-GPU/TPU TYPE: bool DEFAULT: False
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'KR'

Source code in deel/lip/losses.py

def __init__(self, multi_gpu=False, reduction=Reduction.AUTO, name="KR"):
    r"""
    Loss to estimate Wasserstein-1 distance using Kantorovich-Rubinstein duality.
    The Kantorovich-Rubinstein duality is formulated as following:

    $$
    W_1(\mu, \nu) =
    \sup_{f \in Lip_1(\Omega)} \underset{\textbf{x} \sim \mu}{\mathbb{E}}
    \left[f(\textbf{x} )\right] -
    \underset{\textbf{x}  \sim \nu}{\mathbb{E}} \left[f(\textbf{x} )\right]
    $$

    Where mu and nu stands for the two distributions, the distribution where the
    label is 1 and the rest.

    Note that `y_true` and `y_pred` must be of rank 2: (batch_size, 1) or
    (batch_size, C) for multilabel classification (with C categories).
    `y_true` accepts label values in (0, 1), (-1, 1), or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Using a multi-GPU/TPU strategy requires to set `multi_gpu` to True and to
    pre-process the labels `y_true` with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        multi_gpu (bool): set to True when running on multi-GPU/TPU
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.eps = 1e-7
    self.multi_gpu = multi_gpu
    super(KR, self).__init__(reduction=reduction, name=name)
    if multi_gpu:
        self.kr_function = _kr_multi_gpu
    else:
        self.kr_function = partial(_kr, epsilon=self.eps)

MultiMargin

MultiMargin(
    min_margin=1.0,
    reduction=Reduction.AUTO,
    name="MultiMargin",
)

Bases: Loss

Compute the hinge margin loss for multiclass (equivalent to Pytorch multi_margin_loss)

Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
min_margin	margin to enforce. TYPE: float DEFAULT: 1.0
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'MultiMargin'

Source code in deel/lip/losses.py

def __init__(self, min_margin=1.0, reduction=Reduction.AUTO, name="MultiMargin"):
    """
    Compute the hinge margin loss for multiclass (equivalent to Pytorch
    multi_margin_loss)

    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        min_margin (float): margin to enforce.
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    super(MultiMargin, self).__init__(reduction=reduction, name=name)

MulticlassHKR

MulticlassHKR(
    alpha=10.0,
    min_margin=1.0,
    multi_gpu=False,
    reduction=Reduction.AUTO,
    name="MulticlassHKR",
)

Bases: Loss

The multiclass version of HKR. This is done by computing the HKR term over each class and averaging the results.

Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

Using a multi-GPU/TPU strategy requires to set multi_gpu to True and to pre-process the labels y_true with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
alpha	regularization factor TYPE: float DEFAULT: 10.0
min_margin	margin to enforce. TYPE: float DEFAULT: 1.0
multi_gpu	set to True when running on multi-GPU/TPU TYPE: bool DEFAULT: False
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'MulticlassHKR'

Source code in deel/lip/losses.py

def __init__(
    self,
    alpha=10.0,
    min_margin=1.0,
    multi_gpu=False,
    reduction=Reduction.AUTO,
    name="MulticlassHKR",
):
    """
    The multiclass version of HKR. This is done by computing the HKR term over each
    class and averaging the results.

    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Using a multi-GPU/TPU strategy requires to set `multi_gpu` to True and to
    pre-process the labels `y_true` with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        alpha (float): regularization factor
        min_margin (float): margin to enforce.
        multi_gpu (bool): set to True when running on multi-GPU/TPU
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.alpha = tf.Variable(alpha, dtype=tf.float32)
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    self.multi_gpu = multi_gpu
    self.KRloss = MulticlassKR(multi_gpu=multi_gpu, reduction=reduction, name=name)
    if alpha == np.inf:  # alpha = inf => hinge only
        self.fct = partial(multiclass_hinge, min_margin=self.min_margin)
    else:
        self.fct = self.hkr
    super(MulticlassHKR, self).__init__(reduction=reduction, name=name)

MulticlassHinge

MulticlassHinge(
    min_margin=1.0,
    reduction=Reduction.AUTO,
    name="MulticlassHinge",
)

Bases: Loss

Loss to estimate the Hinge loss in a multiclass setup. It computes the element-wise Hinge term. Note that this formulation differs from the one commonly found in tensorflow/pytorch (which maximises the difference between the two largest logits). This formulation is consistent with the binary classification loss used in a multiclass fashion.

Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
min_margin	margin to enforce. TYPE: float DEFAULT: 1.0
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'MulticlassHinge'

Source code in deel/lip/losses.py

def __init__(
    self, min_margin=1.0, reduction=Reduction.AUTO, name="MulticlassHinge"
):
    """
    Loss to estimate the Hinge loss in a multiclass setup. It computes the
    element-wise Hinge term. Note that this formulation differs from the one
    commonly found in tensorflow/pytorch (which maximises the difference between
    the two largest logits). This formulation is consistent with the binary
    classification loss used in a multiclass fashion.

    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        min_margin (float): margin to enforce.
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.min_margin = tf.Variable(min_margin, dtype=tf.float32)
    super(MulticlassHinge, self).__init__(reduction=reduction, name=name)

MulticlassKR

MulticlassKR(
    multi_gpu=False,
    reduction=Reduction.AUTO,
    name="MulticlassKR",
)

Bases: Loss

Loss to estimate average of Wasserstein-1 distance using Kantorovich-Rubinstein duality over outputs. In this multiclass setup, the KR term is computed for each class and then averaged.

Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

Using a multi-GPU/TPU strategy requires to set multi_gpu to True and to pre-process the labels y_true with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
multi_gpu	set to True when running on multi-GPU/TPU TYPE: bool DEFAULT: False
reduction	passed to tf.keras.Loss constructor DEFAULT: AUTO
name	passed to tf.keras.Loss constructor TYPE: str DEFAULT: 'MulticlassKR'

Source code in deel/lip/losses.py

def __init__(self, multi_gpu=False, reduction=Reduction.AUTO, name="MulticlassKR"):
    r"""
    Loss to estimate average of Wasserstein-1 distance using Kantorovich-Rubinstein
    duality over outputs. In this multiclass setup, the KR term is computed for each
    class and then averaged.

    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Using a multi-GPU/TPU strategy requires to set `multi_gpu` to True and to
    pre-process the labels `y_true` with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        multi_gpu (bool): set to True when running on multi-GPU/TPU
        reduction: passed to tf.keras.Loss constructor
        name (str): passed to tf.keras.Loss constructor

    """
    self.eps = 1e-7
    self.multi_gpu = multi_gpu
    super(MulticlassKR, self).__init__(reduction=reduction, name=name)
    if multi_gpu:
        self.kr_function = _kr_multi_gpu
    else:
        self.kr_function = partial(_kr, epsilon=self.eps)

TauBinaryCrossentropy

TauBinaryCrossentropy(
    tau,
    reduction=Reduction.AUTO,
    name="TauBinaryCrossentropy",
)

Bases: Loss

Similar to the original binary crossentropy, but with a settable temperature parameter. y_pred must be a logits tensor (before sigmoid) and not probabilities.

Note that y_true and y_pred must be of rank 2: (batch_size, 1). y_true accepts label values in (0, 1) or (-1, 1).

PARAMETER	DESCRIPTION
tau	temperature parameter.
reduction	reduction of the loss, passed to original loss. DEFAULT: AUTO
name	name of the loss DEFAULT: 'TauBinaryCrossentropy'

Source code in deel/lip/losses.py

def __init__(self, tau, reduction=Reduction.AUTO, name="TauBinaryCrossentropy"):
    """
    Similar to the original binary crossentropy, but with a settable temperature
    parameter. y_pred must be a logits tensor (before sigmoid) and not
    probabilities.

    Note that `y_true` and `y_pred` must be of rank 2: (batch_size, 1). `y_true`
    accepts label values in (0, 1) or (-1, 1).

    Args:
        tau: temperature parameter.
        reduction: reduction of the loss, passed to original loss.
        name: name of the loss
    """
    self.tau = tf.Variable(tau, dtype=tf.float32)
    super().__init__(name=name, reduction=reduction)

TauCategoricalCrossentropy

TauCategoricalCrossentropy(
    tau,
    reduction=Reduction.AUTO,
    name="TauCategoricalCrossentropy",
)

Bases: Loss

Similar to original categorical crossentropy, but with a settable temperature parameter.

PARAMETER	DESCRIPTION
tau	temperature parameter. TYPE: float
reduction	reduction of the loss, passed to original loss. DEFAULT: AUTO
name	name of the loss TYPE: str DEFAULT: 'TauCategoricalCrossentropy'

Source code in deel/lip/losses.py

def __init__(
    self, tau, reduction=Reduction.AUTO, name="TauCategoricalCrossentropy"
):
    """
    Similar to original categorical crossentropy, but with a settable temperature
    parameter.

    Args:
        tau (float): temperature parameter.
        reduction: reduction of the loss, passed to original loss.
        name (str): name of the loss
    """
    self.tau = tf.Variable(tau, dtype=tf.float32)
    super(TauCategoricalCrossentropy, self).__init__(name=name, reduction=reduction)

TauSparseCategoricalCrossentropy

TauSparseCategoricalCrossentropy(
    tau,
    reduction=Reduction.AUTO,
    name="TauSparseCategoricalCrossentropy",
)

Bases: Loss

Similar to original sparse categorical crossentropy, but with a settable temperature parameter.

PARAMETER	DESCRIPTION
tau	temperature parameter. TYPE: float
reduction	reduction of the loss, passed to original loss. DEFAULT: AUTO
name	name of the loss TYPE: str DEFAULT: 'TauSparseCategoricalCrossentropy'

Source code in deel/lip/losses.py

def __init__(
    self, tau, reduction=Reduction.AUTO, name="TauSparseCategoricalCrossentropy"
):
    """
    Similar to original sparse categorical crossentropy, but with a settable
    temperature parameter.

    Args:
        tau (float): temperature parameter.
        reduction: reduction of the loss, passed to original loss.
        name (str): name of the loss
    """
    self.tau = tf.Variable(tau, dtype=tf.float32)
    super().__init__(name=name, reduction=reduction)

hinge_margin

hinge_margin(y_true, y_pred, min_margin)

Compute the element-wise binary hinge margin loss.

Note that y_true and y_pred must be of rank 2: (batch_size, 1) or (batch_size, C) for multilabel classification (with C categories). y_true accepts label values in (0, 1), (-1, 1), or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
min_margin	margin to enforce. TYPE: float

RETURNS	DESCRIPTION
	tf.Tensor: Element-wise hinge margin loss value.

Source code in deel/lip/losses.py

def hinge_margin(y_true, y_pred, min_margin):
    """Compute the element-wise binary hinge margin loss.

    Note that `y_true` and `y_pred` must be of rank 2: (batch_size, 1) or
    (batch_size, C) for multilabel classification (with C categories).
    `y_true` accepts label values in (0, 1), (-1, 1), or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        min_margin (float): margin to enforce.

    Returns:
        tf.Tensor: Element-wise hinge margin loss value.

    """
    sign = tf.where(y_true > 0, 1, -1)
    sign = tf.cast(sign, y_pred.dtype)
    hinge = tf.nn.relu(min_margin / 2.0 - sign * y_pred)
    # In binary case (`y_true` of shape (batch_size, 1)), `tf.reduce_mean(axis=-1)`
    # behaves like `tf.squeeze` to return element-wise loss of shape (batch_size, ).
    return tf.reduce_mean(hinge, axis=-1)

multiclass_hinge

multiclass_hinge(y_true, y_pred, min_margin)

Compute the multi-class hinge margin loss.

y_true and y_pred must be of shape (batch_size, # classes). Note that y_true should be one-hot encoded or pre-processed with the deel.lip.utils.process_labels_for_multi_gpu() function.

PARAMETER	DESCRIPTION
y_true	tensor of true targets of shape (batch_size, # classes) TYPE: Tensor
y_pred	tensor of predicted targets of shape (batch_size, # classes) TYPE: Tensor
min_margin	margin to enforce. TYPE: float

RETURNS	DESCRIPTION
	tf.Tensor: Element-wise multi-class hinge margin loss value.

Source code in deel/lip/losses.py

def multiclass_hinge(y_true, y_pred, min_margin):
    """Compute the multi-class hinge margin loss.

    `y_true` and `y_pred` must be of shape (batch_size, # classes).
    Note that `y_true` should be one-hot encoded or pre-processed with the
    `deel.lip.utils.process_labels_for_multi_gpu()` function.

    Args:
        y_true (tf.Tensor): tensor of true targets of shape (batch_size, # classes)
        y_pred (tf.Tensor): tensor of predicted targets of shape (batch_size, # classes)
        min_margin (float): margin to enforce.

    Returns:
        tf.Tensor: Element-wise multi-class hinge margin loss value.
    """
    sign = tf.where(y_true > 0, 1, -1)
    sign = tf.cast(sign, y_pred.dtype)
    # compute the elementwise hinge term
    hinge = tf.nn.relu(min_margin / 2.0 - sign * y_pred)
    # reweight positive elements
    factor = y_pred.shape[-1] - 1.0
    hinge = tf.where(sign > 0, hinge * factor, hinge)
    return tf.reduce_mean(hinge, axis=-1)