Nonconformity scores

This module provides nonconformity scores for conformal prediction. To be used when building a calibrator.

nonconformity_scores.absolute_difference(y_pred, y_true)

Absolute Deviation.

\[R = |y_{\text{true}}-y_{\text{pred}}|\]

Parameters:

• y_pred (Iterable) – predictions.

• y_true (Iterable) – true labels.

Returns:

mean absolute deviation.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.classwise_lac_score(Y_pred, y_true)

Classwise LAC nonconformity score.

Computes nonconformity scores for classwise conformal prediction. For each sample, the score is stored only for its true class, with NaN values for other classes. This allows computing per-class quantiles during calibration.

Parameters:

• Y_pred (Iterable) – \(Y_{\text{pred}} = (P_{\text{C}_1}, ..., P_{\text{C}_n})\) where \(P_{\text{C}_i}\) is logit associated to class i.

• y_true (Iterable) – true labels.

Returns:

Classwise LAC nonconformity scores of shape (n_samples, n_classes). Entry [i, c] contains 1 - Y_pred[i, c] if y_true[i] == c, else NaN.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.cqr_score(Y_pred, y_true)

CQR nonconformity score. Considering \(Y_{\text{pred}} = (q_{\text{lo}}, q_{\text{hi}})\):

\[R = max\{q_{\text{lo}} - y_{\text{true}}, y_{\text{true}} - q_{\text{hi}}\}\]

where \(q_{\text{lo}}\) (resp. \(q_{\text{hi}}\)) is the lower (resp. higher) quantile prediction

Parameters:

• Y_pred (Iterable) – predicted quantiles couples.

• y_true (Iterable) – true quantiles couples.

Returns:

CQR nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.difference(y_pred, y_true)

Elementwise difference.

\[R = y_{\text{pred}}-y_{\text{true}}\]

Parameters:

• y_pred (Iterable) – predictions.

• y_true (Iterable) – true labels.

Returns:

coordinatewise difference.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.lac_score(Y_pred, y_true)

LAC nonconformity score.

Parameters:

• Y_pred (Iterable) – \(Y_{\text{pred}} = (P_{\text{C}_1}, ..., P_{\text{C}_n})\) where \(P_{\text{C}_i}\) is logit associated to class i.

• y_true (Iterable) – true labels.

Returns:

LAC nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.raps_score(Y_pred, y_true, lambd=0, k_reg=1, rand=True)

RAPS nonconformity score. :rtype: Iterable

Warning

This signature is incompatible with the interface of calibrators. Use raps_score_builder() to properly initialize deel.puncc.api.calibration.BaseCalibrator.

Backend-agnostic implementation relying on backend abstractions.

nonconformity_scores.raps_score_builder(lambd=0, k_reg=1, rand=True)

RAPS nonconformity score builder. When called, returns a RAPS nonconformity score function raps_score() with given initialitation of regularization hyperparameters.

Parameters:

• lambd (float) – positive weight associated to the regularization term that encourages small set sizes. If \(\lambda = 0\), there is no regularization and the implementation identifies with APS.

• k_reg (float) – class rank (ordered by descending probability) starting from which the regularization is applied. For example, if \(k_{reg} = 3\), then the fourth most likely estimated class has an extra penalty of size \(\lambda\).

• rand (bool) – turn on or off the randomization term that smoothes the discrete probability mass jump when including a new class.

Returns:

RAPS nonconformity score function that takes two parameters: Y_pred and y_true.

Return type:

Callable

Raises:

TypeError – unsupported data types.

nonconformity_scores.scaled_ad(Y_pred, y_true, eps=1e-12)

Scaled Absolute Deviation, normalized by an estimation of the conditional mean absolute deviation (conditional MAD). Considering \(Y_{\text{pred}} = (\mu_{\text{pred}}, \sigma_{\text{pred}})\):

\[R = \frac{|y_{\text{true}}-\mu_{\text{pred}}|}{\sigma_{\text{pred}}}\]

Parameters:

• Y_pred (Iterable) – point and conditional MAD predictions. \(Y_{\text{pred}}=(y_{\text{pred}}, \sigma_{\text{pred}})\)

• y_true (Iterable) – true labels.

• eps (float) – small positive value to avoid division by negative or zero.

Returns:

scaled absolute deviation.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.scaled_bbox_difference(Y_pred, Y_true)

Object detection scaled nonconformity score. Considering \(Y_{\text{pred}} = (\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2)\) and \(Y_{\text{true}} = (x_1, y_1, x_2, y_2)\):

\[R = (\frac{\hat{x}_1-x_1}{\Delta x}, \frac{\hat{y}_1-y_1}{\Delta y}, \frac{\hat{x}_2-x_2}{\Delta x}, \frac{\hat{y}_1-y_1}{\Delta y})\]

where \(\Delta x = |\hat{x}_2 - \hat{x}_1|\) and \(\Delta y = |\hat{y}_2 - \hat{y}_1|\).

Parameters:

• Y_pred (Iterable) – predicted coordinates of the bounding box.

• y_true (Iterable) – true coordinates of the bounding box.

Returns:

scaled object detection nonconformity scores.

Return type:

Iterable

Raises:

TypeError – unsupported data types.

nonconformity_scores.weighted_scaled_ad(X, Y_pred, y_true, weight_func, eps=1e-12)

Weighted Scaled Absolute Deviation. Similar to scaled_ad() but with an extra weighting of the nonconformity scores by a function of X. This allows to give more importance to calibration points, as measured by weight function. Considering \(Y_{\text{pred}} = (\mu_{\text{pred}}, \sigma_{\text{pred}})\):

\[ \begin{align}\begin{aligned}R = \frac{|y_{\text{true}}-\mu_{\text{pred}}|}{\sigma_{\text{pred}}} * w(X)\\where :math:`w(X)` is the weight function applied to the inputs.\end{aligned}\end{align} \]

Parameters:

• Y_pred (Iterable) – point predictions. Optionally with conditional MAD predictions if available.

• y_true (Iterable) – true labels.

• weight_func (Callable) – function of X that computes the weighted scores.

• eps (float) – small positive value to avoid division by negative or zero.

Returns:

weighted scaled absolute deviation.

Return type:

Iterable

Raises:

TypeError – unsupported data types.