Nonconformity scores
This module provides nonconformity scores for conformal prediction. To be used when building a calibrator.
- nonconformity_scores.absolute_difference(y_pred, y_true)
Absolute Deviation.
\[R = |y_{\text{true}}-y_{\text{pred}}|\]- Parameters:
y_pred (Iterable) – predictions.
y_true (Iterable) – true labels.
- Returns:
mean absolute deviation.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.cqr_score(Y_pred, y_true)
CQR nonconformity score. Considering \(Y_{\text{pred}} = (q_{\text{lo}}, q_{\text{hi}})\):
\[R = max\{q_{\text{lo}} - y_{\text{true}}, y_{\text{true}} - q_{\text{hi}}\}\]where \(q_{\text{lo}}\) (resp. \(q_{\text{hi}}\)) is the lower (resp. higher) quantile prediction
- Parameters:
Y_pred (Iterable) – predicted quantiles couples.
y_true (Iterable) – true quantiles couples.
- Returns:
CQR nonconformity scores.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.difference(y_pred, y_true)
Elementwise difference.
\[R = y_{\text{pred}}-y_{\text{true}}\]- Parameters:
y_pred (Iterable) – predictions.
y_true (Iterable) – true labels.
- Returns:
coordinatewise difference.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.raps_score(Y_pred, y_true, lambd=0, k_reg=1, rand=True)
RAPS nonconformity score.
Warning
This signature is incompatible with the interface of calibrators. Use
raps_score_builder()
to properly initializedeel.puncc.api.calibration.BaseCalibrator
.- Parameters:
Y_pred (Iterable) – \(Y_{\text{pred}} = (P_{\text{C}_1}, ..., P_{\text{C}_n})\) where \(P_{\text{C}_i}\) is logit associated to class i.
y_true (Iterable) – true labels.
lambd (float) – positive weight associated to the regularization term that encourages small set sizes. If \(\lambda = 0\), there is no regularization and the implementation identifies with APS.
k_reg (float) – class rank (ordered by descending probability) starting from which the regularization is applied. For example, if \(k_{reg} = 3\), then the fourth most likely estimated class has an extra penalty of size \(\lambda\).
rand (bool) – turn on or off the randomization term that smoothes the discrete probability mass jump when including a new class.
- Returns:
RAPS nonconformity scores.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.raps_score_builder(lambd=0, k_reg=1, rand=True)
RAPS nonconformity score builder. When called, returns a RAPS nonconformity score function
raps_score()
with given initialitation of regularization hyperparameters.- Parameters:
lambd (float) – positive weight associated to the regularization term that encourages small set sizes. If \(\lambda = 0\), there is no regularization and the implementation identifies with APS.
k_reg (float) – class rank (ordered by descending probability) starting from which the regularization is applied. For example, if \(k_{reg} = 3\), then the fourth most likely estimated class has an extra penalty of size \(\lambda\).
rand (bool) – turn on or off the randomization term that smoothes the discrete probability mass jump when including a new class.
- Returns:
RAPS nonconformity score function that takes two parameters: Y_pred and y_true.
- Return type:
Callable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.scaled_ad(Y_pred, y_true, eps=1e-12)
Scaled Absolute Deviation, normalized by an estimation of the conditional mean absolute deviation (conditional MAD). Considering \(Y_{\text{pred}} = (\mu_{\text{pred}}, \sigma_{\text{pred}})\):
\[R = \frac{|y_{\text{true}}-\mu_{\text{pred}}|}{\sigma_{\text{pred}}}\]- Parameters:
Y_pred (Iterable) – point and conditional MAD predictions. \(Y_{\text{pred}}=(y_{\text{pred}}, \sigma_{\text{pred}})\)
y_true (Iterable) – true labels.
eps (float) – small positive value to avoid division by negative or zero.
- Returns:
scaled absolute deviation.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.
- nonconformity_scores.scaled_bbox_difference(Y_pred, Y_true)
Object detection scaled nonconformity score. Considering \(Y_{\text{pred}} = (\hat{x}_1, \hat{y}_1, \hat{x}_2, \hat{y}_2)\) and \(Y_{\text{true}} = (x_1, y_1, x_2, y_2)\):
\[R = (\frac{\hat{x}_1-x_1}{\Delta x}, \frac{\hat{y}_1-y_1}{\Delta y}, \frac{\hat{x}_2-x_2}{\Delta x}, \frac{\hat{y}_1-y_1}{\Delta y})\]where \(\Delta x = |\hat{x}_2 - \hat{x}_1|\) and \(\Delta y = |\hat{y}_2 - \hat{y}_1|\).
- Parameters:
Y_pred (Iterable) – predicted coordinates of the bounding box.
y_true (Iterable) – true coordinates of the bounding box.
- Returns:
scaled object detection nonconformity scores.
- Return type:
Iterable
- Raises:
TypeError – unsupported data types.