Confidence intervals for cross-validation. The method is asymptotically exact for the so called Test Error as defined by Bayle et al. (2020). For the (expected) risk, the confidence intervals tend to be too liberal.
Parameters
Those from MeasureAbstractCi
, as well as:
variance
::"all-pairs"
or"within-fold"
How to estimate the variance. The results tend to be very similar.
References
Bayle, Pierre, Bayle, Alexandre, Janson, Lucas, Mackey, Lester (2020). “Cross-validation confidence intervals for test error.” Advances in Neural Information Processing Systems, 33, 16339–16350.
Super classes
mlr3::Measure
-> mlr3inferr::MeasureAbstractCi
-> MeasureCiNaiveCV
Methods
Method new()
Creates a new instance of this R6 class.
Usage
MeasureCiNaiveCV$new(measure)
Arguments
measure
(
Measure
orcharacter(1)
)
A measure of ID of a measure.
Examples
m_naivecv = msr("ci.naive_cv", "classif.ce")
m_naivecv
#> <MeasureCiNaiveCV:classif.ce>: Naive CV CI
#> * Packages: mlr3, mlr3measures, mlr3inferr
#> * Range: [0, 1]
#> * Minimize: TRUE
#> * Average: custom
#> * Parameters: variance=all-pairs, alpha=0.05, within_range=TRUE
#> * Properties: primary_iters
#> * Predict type: response
rr = resample(tsk("sonar"), lrn("classif.featureless"), rsmp("cv"))
rr$aggregate(m_naivecv)
#> classif.ce classif.ce.lower classif.ce.upper
#> 0.4663462 0.3985507 0.5341416