📏 Metrics

This module provides conformal prediction metrics.

metrics.classification_mean_coverage(y_true, set_pred)

Compute empirical coverage of the prediction sets.

Given the \(i\)-th prediction set \(S(X_i)\), the coverage is:

  • \(cov(X_i) = 1\) if \(y_{true} \in S(X_i)\)

  • \(cov(X_i) = 0\) otherwise

With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).

Parameters:
  • y_true (np.ndarray) – Observed label

  • set_pred (Tuple[np.ndarray]) – label prediction set

Returns:

average coverage, indicating the proportion of instances that are correctly covered.

Return type:

float

metrics.classification_mean_size(set_pred)

Compute average size of the prediction sets.

Parameters:

set_pred (Tuple[np.ndarray]) – label prediction set

Returns:

Average size of the prediction sets

Return type:

float

metrics.iou(bboxes1, bboxes2)

Calculates the Intersection over Union (IoU) between two sets of bounding boxes. The IoU is calculated as the ratio between the area of intersection and the area of union between two bounding boxes.

Parameters:
  • bboxes1 (np.ndarray) – array of shape (N, 4) representing the coordinates of N bounding boxes in the format [x_min, y_min, x_max, y_max].

  • bboxes2 (np.ndarray) – array of shape (N, 4) representing the coordinates of N bounding boxes in the format [x_min, y_min, x_max, y_max].

Returns:

iou (numpy.ndarray): Array of shape (N, ) representing the IoU between each pair of bounding boxes.

Return type:

np.ndarray

metrics.object_detection_mean_area(y_pred)

Calculate the mean area of object bounding predictions.

Parameters:

y_pred (np.ndarray) – array of predicted bounding boxes with shape (n, 4).

Returns:

average area of the bounding boxes

Return type:

float

metrics.object_detection_mean_coverage(y_pred_outer, y_true)

Calculate the mean coverage of conformal object detection predictions. For each instance, coverage is defined as the true bounding box being inside the predicted outer bounding box.

Parameters:
  • y_pred (np.ndarray) – array of predicted outer bounding boxes with shape (n, 4).

  • y_true (np.ndarray) – array of true bounding boxes with shape (n, 4).

Returns:

average coverage, indicating the proportion of objects that are correctly covered.

Return type:

float

metrics.regression_ace(y_true, y_pred_lower, y_pred_upper, alpha)

Compte the Average Coverage Error (ACE).

Parameters:
  • y_true (ndarray) – label true values.

  • y_pred_lower (ndarray) – lower bounds of the prediction intervals.

  • y_pred_upper (ndarray) – upper bounds of the prediction intervals.

  • alpha (float) – significance level (max miscoverage target).

Note

The ACE is the distance between the nominal coverage \(1-\alpha\) and the empirical average coverage \(AC\) such that \(ACE = AC - (1-\alpha)\).

If the ACE is strictly negative, the prediction intervals are marginally undercovering. If the ACE is strictly positive, the prediction intervals are maginally conservative.

Returns:

the average coverage error (ACE).

Return type:

float

metrics.regression_mean_coverage(y_true, y_pred_lower, y_pred_upper)

Compute average coverage on several prediction intervals.

Given the \(i\)-th prediction interval \(C(X_i)\), the coverage is:

  • \(cov(X_i) = 1\) if \(y_{true} \in C(X_i)\)

  • \(cov(X_i) = 0\) otherwise

With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).

Parameters:
  • y_true (ndarray) – label true values.

  • y_pred_lower (ndarray) – lower bounds of the prediction intervals.

  • y_pred_upper (ndarray) – upper bounds of the prediction intervals.

Returns:

average coverage, indicating the proportion of instances that are correctly covered.

Return type:

float

metrics.regression_sharpness(y_pred_lower, y_pred_upper)

Compute the average absolute width of the prediction intervals.

Parameters:
  • y_pred_lower (ndarray) – lower bounds of the prediction intervals.

  • y_pred_upper (ndarray) – upper bounds of the prediction intervals.

Returns:

average absolute width of the prediction intervals.

Return type:

float