deel.torchlip.normalizers module¶

The deel.torchlip.normalizers module contains various functions to normalize weights tensors so that all their singular values equal 1.

Normalizers¶

This module contains computation functions for Bjorck and spectral normalization. This is most for internal use.

deel.torchlip.normalizers.bjorck_normalization(w: Tensor, eps: float = 0.001, beta: float = 0.5, maxiter: int = 15) → Tensor[source]¶

Apply Bjorck normalization on the kernel as per

\begin{array}{l} W_0 = W \\ W_{n + 1} = (1 + \beta) W_{n} - \beta W_n W_n^T W_n \\ \overline{W} = W_{N} \end{array}

where $W$ is the kernel of shape $(C, *)$ , with $C$ the number of channels.

Parameters:

w – Weights to normalize. For the normalization to work properly, the greatest eigen value of w must be approximately 1.
eps (float) – epsilon stopping criterion: norm(wt - wt-1) must be less than eps
beta (float) – beta used in each iteration, must be in the interval ]0, 0.5]
maxiter (int) – maximum number of iterations for the algorithm

Returns:

The weights $\overline{W}$ after Bjorck normalization.

deel.torchlip.normalizers.spectral_normalization(kernel: Tensor, u: Tensor | None = None, eps: float = 0.001, maxiter: int = 10) → Tuple[Tensor, Tensor, Tensor][source]¶

Apply spectral normalization on the given kernel $W$ to set its greatest eigen value to approximately 1.

Note

This function is provided for completeness since torchlip layers use the torch.nn.utils.spectral_norm() hook instead.

Parameters:

kernel – The kernel to normalize.
u – Initialization for the initial singular vector. If None, it will be randomly initialized from a normal distribution and twice as much iterations will be performed.
eps (float, optional) – stopping criterion of the algorithm, when norm(u[t] - u[t-1]) is less than eps. Defaults to DEFAULT_EPS_SPECTRAL.
maxiter (int, optional) – maximum number of iterations for the algorithm. Defaults to DEFAULT_MAXITER_SPECTRAL.

Returns:

The normalized kernel $\overline{W}$ , the right singular vector corresponding to the largest singular value, and the largest singular value before normalization.