deel.lip.initializers

This module contains extra Keras initializers, e.g. SpectralInitializer for 1-Lipschitz matrix initialization. They can be used as kernel initializers in any Keras layer.

SpectralInitializer

SpectralInitializer(
    eps_spectral=DEFAULT_EPS_SPECTRAL,
    eps_bjorck=DEFAULT_EPS_BJORCK,
    beta_bjorck=DEFAULT_BETA_BJORCK,
    k_coef_lip=1.0,
    base_initializer="orthogonal",
)

Bases: Initializer

Initialize a kernel to be 1-lipschitz orthogonal using bjorck normalization.

PARAMETER	DESCRIPTION
eps_spectral	stopping criterion of iterative power method TYPE: float DEFAULT: DEFAULT_EPS_SPECTRAL
eps_bjorck	float greater than 0, stopping criterion of bjorck algorithm, setting it to None disable orthogonalization TYPE: float DEFAULT: DEFAULT_EPS_BJORCK
beta_bjorck	beta parameter of bjorck algorithm TYPE: float DEFAULT: DEFAULT_BETA_BJORCK
base_initializer	method used to generate weights before applying the orthonormalization TYPE: str DEFAULT: 'orthogonal'

Source code in deel/lip/initializers.py

def __init__(
    self,
    eps_spectral=DEFAULT_EPS_SPECTRAL,
    eps_bjorck=DEFAULT_EPS_BJORCK,
    beta_bjorck=DEFAULT_BETA_BJORCK,
    k_coef_lip=1.0,
    base_initializer="orthogonal",
) -> None:
    """
    Initialize a kernel to be 1-lipschitz orthogonal using bjorck
    normalization.

    Args:
        eps_spectral (float): stopping criterion of iterative power method
        eps_bjorck (float): float greater than 0, stopping criterion of
            bjorck algorithm, setting it to None disable orthogonalization
        beta_bjorck (float): beta parameter of bjorck algorithm
        base_initializer (str): method used to generate weights before applying the
            orthonormalization
    """
    self.eps_spectral = eps_spectral
    self.eps_bjorck = eps_bjorck
    self.beta_bjorck = beta_bjorck
    self.k_coef_lip = k_coef_lip
    self.base_initializer = initializers.get(base_initializer)
    super(SpectralInitializer, self).__init__()