Naive Cross-Validation CI

Confidence intervals for cross-validation. The method is asymptotically exact for the so called Test Error as defined by Bayle et al. (2020). For the (expected) risk, the confidence intervals tend to be too liberal.

Parameters

Those from MeasureAbstractCi, as well as:

• variance :: "all-pairs" or "within-fold"
  How to estimate the variance. The results tend to be very similar.

References

Bayle, Pierre, Bayle, Alexandre, Janson, Lucas, Mackey, Lester (2020). “Cross-validation confidence intervals for test error.” Advances in Neural Information Processing Systems, 33, 16339–16350.

Super classes

mlr3::Measure -> mlr3inferr::MeasureAbstractCi -> MeasureCiNaiveCV

Methods

Public methods

• MeasureCiNaiveCV$new()

• MeasureCiNaiveCV$clone()

Inherited methods

• mlr3::Measure$format()
• mlr3::Measure$help()
• mlr3::Measure$print()
• mlr3::Measure$score()
• mlr3inferr::MeasureAbstractCi$aggregate()

---

Method new()

Creates a new instance of this R6 class.

Usage

MeasureCiNaiveCV$new(measure)

Arguments

measure

(Measure or character(1))
A measure of ID of a measure.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

MeasureCiNaiveCV$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

m_naivecv = msr("ci.naive_cv", "classif.ce")
m_naivecv
#> <MeasureCiNaiveCV:classif.ce>: Naive CV CI
#> * Packages: mlr3, mlr3measures, mlr3inferr
#> * Range: [0, 1]
#> * Minimize: TRUE
#> * Average: custom
#> * Parameters: variance=all-pairs, alpha=0.05, within_range=TRUE
#> * Properties: primary_iters
#> * Predict type: response
rr = resample(tsk("sonar"), lrn("classif.featureless"), rsmp("cv"))
rr$aggregate(m_naivecv)
#>       classif.ce classif.ce.lower classif.ce.upper 
#>        0.4663462        0.3985507        0.5341416