📏 Metrics

This module provides conformal prediction metrics.

metrics.classification_mean_coverage(y_true, set_pred)

Compute empirical coverage of the prediction sets.

Given the \(i\)-th prediction set \(S(X_i)\), the coverage is:

• \(cov(X_i) = 1\) if \(y_{true} \in S(X_i)\)

• \(cov(X_i) = 0\) otherwise

With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).

Parameters:

• y_true (np.ndarray) – Observed label

• set_pred (Tuple[np.ndarray]) – label prediction set

Returns:

average coverage, indicating the proportion of instances that are correctly covered.

Return type:

float

metrics.classification_mean_size(set_pred)

Compute average size of the prediction sets.

Parameters:

set_pred (Tuple[np.ndarray]) – label prediction set

Returns:

Average size of the prediction sets

Return type:

float

metrics.object_detection_mean_area(y_pred)

Calculate the mean area of object bounding predictions.

Parameters:

y_pred (np.ndarray) – array of predicted bounding boxes with shape (n, 4).

Returns:

average area of the bounding boxes

Return type:

float

metrics.object_detection_mean_coverage(y_pred_outer, y_true)

Calculate the mean coverage of conformal object detection predictions. For each instance, coverage is defined as the true bounding box being inside the predicted outer bounding box.

Parameters:

• y_pred (np.ndarray) – array of predicted outer bounding boxes with shape (n, 4).

• y_true (np.ndarray) – array of true bounding boxes with shape (n, 4).

Returns:

average coverage, indicating the proportion of objects that are correctly covered.

Return type:

float

metrics.regression_ace(y_true, y_pred_lower, y_pred_upper, alpha)

Compte the Average Coverage Error (ACE).

Parameters:

• y_true (ndarray) – label true values.

• y_pred_lower (ndarray) – lower bounds of the prediction intervals.

• y_pred_upper (ndarray) – upper bounds of the prediction intervals.

• alpha (float) – significance level (max miscoverage target).

Note

The ACE is the distance between the nominal coverage \(1-\alpha\) and the empirical average coverage \(AC\) such that \(ACE = AC - (1-\alpha)\).

If the ACE is strictly negative, the prediction intervals are marginally undercovering. If the ACE is strictly positive, the prediction intervals are maginally conservative.

Returns:

the average coverage error (ACE).

Return type:

float

metrics.regression_mean_coverage(y_true, y_pred_lower, y_pred_upper)

Compute average coverage on several prediction intervals.

Given the \(i\)-th prediction interval \(C(X_i)\), the coverage is:

• \(cov(X_i) = 1\) if \(y_{true} \in C(X_i)\)

• \(cov(X_i) = 0\) otherwise

With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).

Parameters:

• y_true (ndarray) – label true values.

• y_pred_lower (ndarray) – lower bounds of the prediction intervals.

• y_pred_upper (ndarray) – upper bounds of the prediction intervals.

Returns:

average coverage, indicating the proportion of instances that are correctly covered.

Return type:

float

metrics.regression_sharpness(y_pred_lower, y_pred_upper)

Compute the average absolute width of the prediction intervals.

Parameters:

• y_pred_lower (ndarray) – lower bounds of the prediction intervals.

• y_pred_upper (ndarray) – upper bounds of the prediction intervals.

Returns:

average absolute width of the prediction intervals.

Return type:

float