deel.torchlip.normalizers module¶

The deel.torchlip.normalizers module contains various functions to normalize weights tensors so that all their singular values equal 1.

Normalizers¶

This module contains computation functions for Bjorck and spectral normalization. This is most for internal use.

deel.torchlip.normalizers.bjorck_normalization(w: torch.Tensor, niter: int = 15, beta: float = 0.5) → torch.Tensor[source]¶

Apply Bjorck normalization on the kernel as per

\begin{array}{l} W_0 = W \\ W_{n + 1} = (1 + \beta) W_{n} - \beta W_n W_n^T W_n \\ \overline{W} = W_{N} \end{array}

where $W$ is the kernel of shape $(C, *)$ , with $C$ the number of channels.

Parameters

w – Weights to normalize. For the normalization to work properly, the greatest eigen value of w must be approximately 1.
niter – Number of iterations.
beta – Value of $\beta$ to use.

Returns

The weights $\overline{W}$ after Bjorck normalization.

deel.torchlip.normalizers.spectral_normalization(kernel: torch.Tensor, u: Optional[torch.Tensor] = None, niter: int = 3) → Tuple[torch.Tensor, torch.Tensor, torch.Tensor][source]¶

Apply spectral normalization on the given kernel $W$ to set its greatest eigen value to approximately 1.

Note

This function is provided for completeness since torchlip layers use the torch.nn.utils.spectral_norm() hook instead.

Parameters

kernel – The kernel to normalize.
u – Initialization for the initial singular vector. If None, it will be randomly initialized from a normal distribution and twice as much iterations will be performed.
niter – Number of iteration. If u is not specified, we perform twice as much iterations.

Returns

The normalized kernel $\overline{W}$ , the right singular vector corresponding to the largest singular value, and the largest singular value before normalization.