Nonconformity scores

This module provides nonconformity scores for conformal prediction. To be used when building a calibrator.

nonconformity_scores.absolute_difference(y_pred, y_true)

Absolute Deviation.

\[R = |y_{\text{true}}-y_{\text{pred}}|\]
Parameters:

  • y_pred (Iterable) – predictions.

  • y_true (Iterable) – true labels.

Returns:

mean absolute deviation.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.cqr_score(Y_pred, y_true)

CQR nonconformity score. Considering \(Y_{\text{pred}} = (q_{\text{lo}}, q_{\text{hi}})\):

\[R = max\{q_{\text{lo}} - y_{\text{true}}, y_{\text{true}} - q_{\text{hi}}\}\]

where \(q_{\text{lo}}\) (resp. \(q_{\text{hi}}\)) is the lower (resp. higher) quantile prediction

Parameters:

  • Y_pred (Iterable) – predicted quantiles couples.

  • y_true (Iterable) – true quantiles couples.

Returns:

CQR nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.difference(y_pred, y_true)

Elementwise difference.

\[R = y_{\text{pred}}-y_{\text{true}}\]
Parameters:

  • y_pred (Iterable) – predictions.

  • y_true (Iterable) – true labels.

Returns:

coordinatewise difference.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.lac_score(Y_pred, y_true)

LAC nonconformity score.

Parameters:

  • Y_pred (Iterable) – \(Y_{\text{pred}} = (P_{\text{C}_1}, ..., P_{\text{C}_n})\) where \(P_{\text{C}_i}\) is logit associated to class i.

  • y_true (Iterable) – true labels.

Returns:

RAPS nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.raps_score(Y_pred, y_true, lambd=0, k_reg=1, rand=True)

RAPS nonconformity score.

Warning

This signature is incompatible with the interface of calibrators. Use raps_score_builder() to properly initialize deel.puncc.api.calibration.BaseCalibrator.

Parameters:

  • Y_pred (Iterable) – \(Y_{\text{pred}} = (P_{\text{C}_1}, ..., P_{\text{C}_n})\) where \(P_{\text{C}_i}\) is logit associated to class i.

  • y_true (Iterable) – true labels.

  • lambd (float) – positive weight associated to the regularization term that encourages small set sizes. If \(\lambda = 0\), there is no regularization and the implementation identifies with APS.

  • k_reg (float) – class rank (ordered by descending probability) starting from which the regularization is applied. For example, if \(k_{reg} = 3\), then the fourth most likely estimated class has an extra penalty of size \(\lambda\).

  • rand (bool) – turn on or off the randomization term that smoothes the discrete probability mass jump when including a new class.

Returns:

RAPS nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.raps_score_builder(lambd=0, k_reg=1, rand=True)

RAPS nonconformity score builder. When called, returns a RAPS nonconformity score function raps_score() with given initialitation of regularization hyperparameters.

Parameters:

  • lambd (float) – positive weight associated to the regularization term that encourages small set sizes. If \(\lambda = 0\), there is no regularization and the implementation identifies with APS.

  • k_reg (float) – class rank (ordered by descending probability) starting from which the regularization is applied. For example, if \(k_{reg} = 3\), then the fourth most likely estimated class has an extra penalty of size \(\lambda\).

  • rand (bool) – turn on or off the randomization term that smoothes the discrete probability mass jump when including a new class.

Returns:

RAPS nonconformity score function that takes two parameters: Y_pred and y_true.

Return type:

Callable

Raises:

TypeError – unsupported data types.

nonconformity_scores.scaled_ad(Y_pred, y_true, eps=1e-12)

Scaled Absolute Deviation, normalized by an estimation of the conditional mean absolute deviation (conditional MAD). Considering \(Y_{\text{pred}} = (\mu_{\text{pred}}, \sigma_{\text{pred}})\):

\[R = \frac{|y_{\text{true}}-\mu_{\text{pred}}|}{\sigma_{\text{pred}}}\]
Parameters:

  • Y_pred (Iterable) – point and conditional MAD predictions. \(Y_{\text{pred}}=(y_{\text{pred}}, \sigma_{\text{pred}})\)

  • y_true (Iterable) – true labels.

  • eps (float) – small positive value to avoid division by negative or zero.

Returns:

scaled absolute deviation.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.scaled_bbox_difference(Y_pred, Y_true)

Object detection scaled nonconformity score. Considering \(Y_{\text{pred}} = (\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2)\) and \(Y_{\text{true}} = (x_1, y_1, x_2, y_2)\):

\[R = (\frac{\hat{x}_1-x_1}{\Delta x}, \frac{\hat{y}_1-y_1}{\Delta y}, \frac{\hat{x}_2-x_2}{\Delta x}, \frac{\hat{y}_2-y_2}{\Delta y})\]

where \(\Delta x = |\hat{x}_2 - \hat{x}_1|\) and \(\Delta y = |\hat{y}_2 - \hat{y}_1|\).

Parameters:

  • Y_pred (Iterable) – predicted coordinates of the bounding box.

  • y_true (Iterable) – true coordinates of the bounding box.

Returns:

scaled object detection nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.