Skip to content

deel.lip.constraints

This module contains extra constraint objects. These object can be added as params to regular layers.

AutoWeightClipConstraint 

AutoWeightClipConstraint(scale=1)

Bases: Constraint

Clips the weights incident to each hidden unit to be inside the range [-c,+c]. With c = 1/sqrt(size(kernel)).

PARAMETER DESCRIPTION
scale

scaling factor to increase/decrease clipping value.

TYPE: float DEFAULT: 1

Source code in deel/lip/constraints.py 
41
42
43
44
45
46
47
48
49
def __init__(self, scale=1):
    """
    Clips the weights incident to each hidden unit to be inside the range `[-c,+c]`.
    With c = 1/sqrt(size(kernel)).

    Args:
        scale (float): scaling factor to increase/decrease clipping value.
    """
    self.scale = scale

FrobeniusConstraint 

FrobeniusConstraint(eps=1e-07)

Bases: Constraint

Constrain the weights by dividing the weight matrix by it's L2 norm.

Source code in deel/lip/constraints.py 
63
64
65
66
67
def __init__(self, eps=1e-7):
    """
    Constrain the weights by dividing the weight matrix by it's L2 norm.
    """
    self.eps = eps

SpectralConstraint 

SpectralConstraint(
    k_coef_lip=1.0,
    eps_spectral=DEFAULT_EPS_SPECTRAL,
    eps_bjorck=DEFAULT_EPS_BJORCK,
    beta_bjorck=DEFAULT_BETA_BJORCK,
    u=None,
)

Bases: Constraint

Ensure that all singular values of the weight matrix equals to 1. Computation based on Bjorck algorithm. The computation is done in two steps:

  1. reduce the larget singular value to k_coef_lip, using iterate power method.
  2. increase other singular values to k_coef_lip, using bjorck algorithm.
PARAMETER DESCRIPTION
k_coef_lip

lipschitz coefficient of the weight matrix

TYPE: float DEFAULT: 1.0

eps_spectral

stopping criterion for the iterative power algorithm.

TYPE: float DEFAULT: DEFAULT_EPS_SPECTRAL

eps_bjorck

stopping criterion Bjorck algorithm.

TYPE: float DEFAULT: DEFAULT_EPS_BJORCK

beta_bjorck

beta parameter in bjorck algorithm.

TYPE: float DEFAULT: DEFAULT_BETA_BJORCK

u

vector used for iterated power method, can be set to None (used for serialization/deserialization purposes).

TYPE: Tensor DEFAULT: None

Source code in deel/lip/constraints.py 
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
def __init__(
    self,
    k_coef_lip=1.0,
    eps_spectral=DEFAULT_EPS_SPECTRAL,
    eps_bjorck=DEFAULT_EPS_BJORCK,
    beta_bjorck=DEFAULT_BETA_BJORCK,
    u=None,
) -> None:
    """
    Ensure that *all* singular values of the weight matrix equals to 1. Computation
    based on Bjorck algorithm. The computation is done in two steps:

    1. reduce the larget singular value to k_coef_lip, using iterate power method.
    2. increase other singular values to k_coef_lip, using bjorck algorithm.

    Args:
        k_coef_lip (float): lipschitz coefficient of the weight matrix
        eps_spectral (float): stopping criterion for the iterative power algorithm.
        eps_bjorck (float): stopping criterion Bjorck algorithm.
        beta_bjorck (float): beta parameter in bjorck algorithm.
        u (tf.Tensor): vector used for iterated power method, can be set to None
            (used for serialization/deserialization purposes).
    """
    self.eps_spectral = eps_spectral
    self.eps_bjorck = eps_bjorck
    self.beta_bjorck = beta_bjorck
    self.k_coef_lip = k_coef_lip
    if not (isinstance(u, tf.Tensor) or (u is None)):
        u = tf.convert_to_tensor(u)
    self.u = u
    super(SpectralConstraint, self).__init__()

WeightClipConstraint 

WeightClipConstraint(c=2)

Bases: Constraint

Clips the weights incident to each hidden unit to be inside the range [-c,+c].

PARAMETER DESCRIPTION
c

clipping parameter.

TYPE: float DEFAULT: 2

Source code in deel/lip/constraints.py 
23
24
25
26
27
28
29
30
def __init__(self, c=2):
    """
    Clips the weights incident to each hidden unit to be inside the range `[-c,+c]`.

    Args:
        c (float): clipping parameter.
    """
    self.c = c