📏 Metrics
This module provides conformal prediction metrics.
- metrics.classification_mean_coverage(y_true, set_pred)
Compute empirical coverage of the prediction sets.
Given the \(i\)-th prediction set \(S(X_i)\), the coverage is:
\(cov(X_i) = 1\) if \(y_{true} \in S(X_i)\)
\(cov(X_i) = 0\) otherwise
With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).
- Parameters:
y_true (np.ndarray) – Observed label
set_pred (Tuple[np.ndarray]) – label prediction set
- Returns:
average coverage, indicating the proportion of instances that are correctly covered.
- Return type:
float
- metrics.classification_mean_size(set_pred)
Compute average size of the prediction sets.
- Parameters:
set_pred (Tuple[np.ndarray]) – label prediction set
- Returns:
Average size of the prediction sets
- Return type:
float
- metrics.iou(bboxes1, bboxes2)
Calculates the Intersection over Union (IoU) between two sets of bounding boxes. The IoU is calculated as the ratio between the area of intersection and the area of union between two bounding boxes.
- Parameters:
bboxes1 (np.ndarray) – array of shape (N, 4) representing the coordinates of N bounding boxes in the format [x_min, y_min, x_max, y_max].
bboxes2 (np.ndarray) – array of shape (N, 4) representing the coordinates of N bounding boxes in the format [x_min, y_min, x_max, y_max].
- Returns:
iou (numpy.ndarray): Array of shape (N, ) representing the IoU between each pair of bounding boxes.
- Return type:
np.ndarray
- metrics.object_detection_mean_area(y_pred)
Calculate the mean area of object bounding predictions.
- Parameters:
y_pred (np.ndarray) – array of predicted bounding boxes with shape (n, 4).
- Returns:
average area of the bounding boxes
- Return type:
float
- metrics.object_detection_mean_coverage(y_pred_outer, y_true)
Calculate the mean coverage of conformal object detection predictions. For each instance, coverage is defined as the true bounding box being inside the predicted outer bounding box.
- Parameters:
y_pred (np.ndarray) – array of predicted outer bounding boxes with shape (n, 4).
y_true (np.ndarray) – array of true bounding boxes with shape (n, 4).
- Returns:
average coverage, indicating the proportion of objects that are correctly covered.
- Return type:
float
- metrics.regression_ace(y_true, y_pred_lower, y_pred_upper, alpha)
Compte the Average Coverage Error (ACE).
- Parameters:
y_true (ndarray) – label true values.
y_pred_lower (ndarray) – lower bounds of the prediction intervals.
y_pred_upper (ndarray) – upper bounds of the prediction intervals.
alpha (float) – significance level (max miscoverage target).
Note
The ACE is the distance between the nominal coverage \(1-\alpha\) and the empirical average coverage \(AC\) such that \(ACE = AC - (1-\alpha)\).
If the ACE is strictly negative, the prediction intervals are marginally undercovering. If the ACE is strictly positive, the prediction intervals are maginally conservative.
- Returns:
the average coverage error (ACE).
- Return type:
float
- metrics.regression_mean_coverage(y_true, y_pred_lower, y_pred_upper)
Compute average coverage on several prediction intervals.
Given the \(i\)-th prediction interval \(C(X_i)\), the coverage is:
\(cov(X_i) = 1\) if \(y_{true} \in C(X_i)\)
\(cov(X_i) = 0\) otherwise
With N the number of examples, the average coverage is \(1/N \sum_{i=1}^{N} cov(X_i)\).
- Parameters:
y_true (ndarray) – label true values.
y_pred_lower (ndarray) – lower bounds of the prediction intervals.
y_pred_upper (ndarray) – upper bounds of the prediction intervals.
- Returns:
average coverage, indicating the proportion of instances that are correctly covered.
- Return type:
float
- metrics.regression_sharpness(y_pred_lower, y_pred_upper)
Compute the average absolute width of the prediction intervals.
- Parameters:
y_pred_lower (ndarray) – lower bounds of the prediction intervals.
y_pred_upper (ndarray) – upper bounds of the prediction intervals.
- Returns:
average absolute width of the prediction intervals.
- Return type:
float