Performance metrics

The skpro.metrics module contains metrics for evaluating probabilistic predictions, including survival and time-to-event predictions.

All metrics in skpro can be listed using the skpro.registry.all_objects utility, using object_types="metric", optionally filtered by tags. Valid tags can be listed using sktime.registry.all_tags.

Survival/time-to-event specific metrics in skpro can be listed by filtering by capability:survival being True.

All probabilistic metrics can be used for survival prediction, by default they will ignore the censoring information. Note: this is different from subsetting to uncensored observations.

Quantile and interval prediction metrics

PinballLoss([multioutput, score_average, alpha])	Evaluate the pinball loss at all quantiles given in data.
EmpiricalCoverage([multioutput, score_average])	Evaluate the pinball loss at all quantiles given in data.
ConstraintViolation([multioutput, score_average])	Evaluate the pinball loss at all quantiles given in data.

Distribution prediction metrics

Distribution predictions are also known as conditional distribution predictions. (or conditional density predictions, if continuous).

CRPS([multioutput, multivariate])	Continuous rank probability score for distributional predictions.
LogLoss([multioutput, multivariate])	Logarithmic loss for distributional predictions.
SquaredDistrLoss([multioutput, multivariate])	Squared loss for distributional predictions.
LinearizedLogLoss([range, multioutput, ...])	Lineararized logarithmic loss for distributional predictions.
SquaredDistrLoss([multioutput, multivariate])	Squared loss for distributional predictions.

Survival prediction metrics

Survival or time-to-event predictions are a variant of distribution predictions, where the ground truth may be censored. These metrics take the censoring information into account.

ConcordanceHarrell([score, score_args, ...])	Concordance index (Harrell).
SPLL([multioutput, multivariate])	Survival Process Logarithmic Loss for distributional predictions.