Reference

A container listing all available methods for conformal prediction.

A container listing all atomic MLJ models that have been tested for use with this package.

conformal_model(model::Supervised; method::Union{Nothing, Symbol}=nothing, kwargs...)

A simple wrapper function that turns a model::Supervised into a conformal model. It accepts an optional key argument that can be used to specify the desired method for conformal prediction as well as additinal kwargs... specific to the method.

emp_coverage(ŷ, y)

Computes the empirical coverage for conformal predictions ŷ.

ineff(ŷ)

Computes the inefficiency (average set size) for conformal predictions ŷ.

partial_fit(conf_model::TimeSeriesRegressorEnsembleBatch, fitresult, X, y, shift_size)

For the TimeSeriesRegressorEnsembleBatch Non-conformity scores are updated by the most recent data (X,y). shift_size determines how many points in Non-conformity scores will be discarded.

set_size(ŷ)

Helper function that computes the set size for conformal predictions.

size_stratified_coverage(ŷ, y)

Computes the size-stratified coverage for conformal predictions ŷ.

MMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)

For the AdaptiveInductiveClassifier nonconformity scores are computed by cumulatively summing the ranked scores of each label in descending order until reaching the true label $Y_i$:

$S_i^{\text{CAL}} = s(X_i,Y_i) = \sum_{j=1}^k \hat\mu(X_i)_{\pi_j} \ \text{where } \ Y_i=\pi_k, i \in \mathcal{D}_{\text{calibration}}$

MMI.fit(conf_model::CVMinMaxRegressor, verbosity, X, y)

For the CVMinMaxRegressor nonconformity scores are computed in the same way as for the CVPlusRegressor. Specifically, we have,

$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::CVPlusRegressor, verbosity, X, y)

For the CVPlusRegressor nonconformity scores are computed though cross-validation (CV) as follows,

$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::ConformalQuantileRegressor, verbosity, X, y)

For the ConformalQuantileRegressor nonconformity scores are computed as follows:

$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu_{\alpha_{lo}}(X_i), \hat\mu_{\alpha_{hi}}(X_i) ,Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu_{\alpha_{lo}}(X_i), \hat\mu_{\alpha_{hi}}(X_i) ,Y_i)= max\{\hat\mu_{\alpha_{low}}(X_i)-Y_i, Y_i-\hat\mu_{\alpha_{hi}}(X_i)\}$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}` and$\alpha{lo}, \alpha{hi}`` lower and higher percentile.

MMI.fit(conf_model::JackknifeMinMaxRegressor, verbosity, X, y)

For the JackknifeMinMaxRegressor nonconformity scores are computed in the same way as for the JackknifeRegressor. Specifically, we have,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusMinMaxAbRegressor, verbosity, X, y)

For the JackknifePlusAbMinMaxRegressor nonconformity scores are as,

$S_i^{\text{J+MinMax}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusAbRegressor, verbosity, X, y)

For the JackknifePlusAbRegressor nonconformity scores are computed as

$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusRegressor, verbosity, X, y)

For the JackknifePlusRegressor nonconformity scores are computed in the same way as for the JackknifeRegressor. Specifically, we have,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifeRegressor, verbosity, X, y)

For the JackknifeRegressor nonconformity scores are computed through a leave-one-out (LOO) procedure as follows,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::NaiveClassifier, verbosity, X, y)

For the NaiveClassifier nonconformity scores are computed in-sample as follows:

$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::NaiveRegressor, verbosity, X, y)

For the NaiveRegressor nonconformity scores are computed in-sample as follows:

$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y)

For the SimpleInductiveClassifier nonconformity scores are computed as follows:

$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$. The simple approach only takes the softmax probability of the true label into account.

MMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y)

For the SimpleInductiveRegressor nonconformity scores are computed as follows:

$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::TimeSeriesRegressorEnsembleBatch, verbosity, X, y)

For the TimeSeriesRegressorEnsembleBatch nonconformity scores are computed as

$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)

For the AdaptiveInductiveClassifier prediction sets are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::CVMinMaxRegressor, fitresult, Xnew)

For the CVMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CV}} \}, \max_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{ S_i^{\text{CV}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with subset $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.

MMI.predict(conf_model::CVPlusRegressor, fitresult, Xnew)

For the CVPlusRegressor prediction intervals are computed in much same way as for the JackknifePlusRegressor. Specifically, we have,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - S_i^{\text{CV}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + S_i^{\text{CV}}\} \right] , \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with fold $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.

The JackknifePlusRegressor is a special case of the CVPlusRegressor for which $K=n$.

MMI.predict(conf_model::ConformalQuantileRegressor, fitresult, Xnew)

For the ConformalQuantileRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = [\hat\mu_{\alpha_{lo}}(X_{n+1}) - \hat{q}_{n, \alpha} \{S_i^{\text{CAL}} \}, \hat\mu_{\alpha_{hi}}(X_{n+1}) + \hat{q}_{n, \alpha} \{S_i^{\text{CAL}} \}], \ i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::JackknifeMinMaxRegressor, fitresult, Xnew)

For the JackknifeMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife-minmax procedure is more conservative than the JackknifePlusRegressor.

MMI.predict(conf_model::JackknifePlusAbMinMaxRegressor, fitresult, Xnew)

For the JackknifePlusAbMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}^{J+MinMax}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife+ab-minmax procedure is more conservative than the JackknifePlusAbRegressor.

MMI.predict(conf_model::JackknifePlusAbRegressor, fitresult, Xnew)

For the JackknifePlusAbRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

MMI.predict(conf_model::JackknifePlusRegressor, fitresult, Xnew)

For the JackknifePlusRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-i}(X_{n+1}) - S_i^{\text{LOO}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-i}(X_{n+1}) + S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

MMI.predict(conf_model::JackknifeRegressor, fitresult, Xnew)

For the JackknifeRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\}, \ i \in \mathcal{D}_{\text{train}}$

where $S_i^{\text{LOO}}$ denotes the nonconformity that is generated as explained in fit(conf_model::JackknifeRegressor, verbosity, X, y). The jackknife procedure addresses the overfitting issue associated with the NaiveRegressor.

MMI.predict(conf_model::NaiveClassifier, fitresult, Xnew)

For the NaiveClassifier prediction sets are computed as follows:

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \} \right\}, \ i \in \mathcal{D}_{\text{train}}$

The naive approach typically produces prediction regions that undercover due to overfitting.

MMI.predict(conf_model::NaiveRegressor, fitresult, Xnew)

For the NaiveRegressor prediction intervals are computed as follows:

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \}, \ i \in \mathcal{D}_{\text{train}}$

The naive approach typically produces prediction regions that undercover due to overfitting.

MMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew)

For the SimpleInductiveClassifier prediction sets are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, \ i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::SimpleInductiveRegressor, fitresult, Xnew)

For the SimpleInductiveRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}} \}, \ i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::TimeSeriesRegressorEnsembleBatch, fitresult, Xnew)

For the TimeSeriesRegressorEnsembleBatch prediction intervals are computed as follows,

$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

soft_assignment(conf_model::ConformalProbabilisticSet, fitresult, X; temp::Real=0.1)

This function can be used to compute soft assigment probabilities for new data X as in soft_assignment(conf_model::ConformalProbabilisticSet; temp::Real=0.1). When a fitted model $\mu$ (fitresult) and new samples X are supplied, non-conformity scores are first computed for the new data points. Then the existing threshold/quantile q̂ is used to compute the final soft assignments.

soft_assignment(conf_model::ConformalProbabilisticSet; temp::Real=0.1)

Computes soft assignment scores for each label and sample. That is, the probability of label k being included in the confidence set. This implementation follows Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. Contrary to the paper, we use non-conformity scores instead of conformity scores, hence the sign swap.

The AdaptiveInductiveClassifier is an improvement to the SimpleInductiveClassifier and the NaiveClassifier. Contrary to the NaiveClassifier it computes nonconformity scores using a designated calibration dataset like the SimpleInductiveClassifier. Contrary to the SimpleInductiveClassifier it utilizes the softmax output of all classes.

Constructor for CVMinMaxRegressor.

Constructor for CVPlusRegressor.

An abstract base type for conformal models that produce interval-valued predictions. This includes most conformal regression models.

An abstract base type for conformal models that produce probabilistic predictions. This includes some conformal classifier like Venn-ABERS.

An abstract base type for conformal models that produce set-valued probabilistic predictions. This includes most conformal classification models.

Constructor for ConformalQuantileRegressor.

Constructor for JackknifeMinMaxRegressor.

Constructor for JackknifePlusAbMinMaxRegressor.

Constructor for JackknifePlusAbPlusRegressor.

Constructor for JackknifePlusRegressor.

Constructor for JackknifeRegressor.

The NaiveClassifier is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier it computes nonconformity scores using a designated training dataset.

The NaiveRegressor for conformal prediction is the simplest approach to conformal regression.

Union type for quantile models.

The SimpleInductiveClassifier is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier it computes nonconformity scores using a designated calibration dataset.

The SimpleInductiveRegressor is the simplest approach to Inductive Conformal Regression. Contrary to the NaiveRegressor it computes nonconformity scores using a designated calibration dataset.

Constructor for TimeSeriesRegressorEnsembleBatch.

_aggregate(y, aggregate::Union{Symbol,String})

Helper function that performs aggregation across vector of predictions.

absolute_error(y,ŷ)

Computes abs(y - ŷ) where ŷ is the predicted value.

blockbootstrap(time_series_data, block_szie)

Generate a sampling method, that block bootstraps the given data

is_classification(ŷ)

Helper function that checks if conformal prediction ŷ comes from a conformal classification model.

is_covered(ŷ, y)

Helper function to check if y is contained in conformal region. Based on whether conformal predictions ŷ are set- or interval-valued, different checks are executed.

is_covered_interval(ŷ, y)

Helper function to check if y is contained in conformal interval.

is_covered_set(ŷ, y)

Helper function to check if y is contained in conformal set.

is_regression(ŷ)

Helper function that checks if conformal prediction ŷ comes from a conformal regression model.

minus_softmax(y,ŷ)

Computes 1.0 - ŷ where ŷ is the softmax output for a given class.

qminus(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{-}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.

qplus(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{+}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.

reformat_interval(ŷ)

Reformats conformal interval predictions.

reformat_mlj_prediction(ŷ)

A helper function that extracts only the output (predicted values) for whatever is returned from MMI.predict(model, fitresult, Xnew). This is currently used to avoid issues when calling MMI.predict(model, fitresult, Xnew) in pipelines.

score(conf_model::AdaptiveInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Score method for the AdaptiveInductiveClassifier dispatched for any <:Supervised model.

score(conf_model::SimpleInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Score method for the SimpleInductiveClassifier dispatched for any <:Supervised model.

score(conf_model::ConformalProbabilisticSet, fitresult, X, y=nothing)

Generic score method for the ConformalProbabilisticSet. It computes nonconformity scores using the heuristic function h and the softmax probabilities of the true class. Method is dispatched for different Conformal Probabilistic Sets and atomic models.

split_data(conf_model::ConformalProbabilisticSet, indices::Base.OneTo{Int})

Splits the data into a proper training and calibration set.

The ConformalNNClassifier struct is a wrapper for a ConformalModel that can be used with MLJFlux.jl.

The ConformalNNRegressor struct is a wrapper for a ConformalModel that can be used with MLJFlux.jl.

classification_loss(
    conf_model::ConformalProbabilisticSet, fitresult, X, y;
    loss_matrix::Union{AbstractMatrix,UniformScaling}=UniformScaling(1.0),
    temp::Real=0.1
)

Computes the calibration loss following Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. Following the notation in the paper, the loss is computed as,

\[\mathcal{L}(C_{\theta}(x;\tau),y) = \sum_k L_{y,k} \left[ (1 - C_{\theta,k}(x;\tau)) \mathbf{I}_{y=k} + C_{\theta,k}(x;\tau) \mathbf{I}_{y\ne k} \right]\]

where $\tau$ is just the quantile q̂ and $\kappa$ is the target set size (defaults to $1$).

qminus_smooth(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{-}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf. To allow for differentiability, we use the soft sort function from InferOpt.jl.

qplus_smooth(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{+}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf. To allow for differentiability, we use the soft sort function from InferOpt.jl.

ConformalPrediction.score(conf_model::AdaptiveInductiveClassifier, ::Type{<:MLJFluxModel}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Overloads the score function for the MLJFluxModel type.

ConformalPrediction.score(conf_model::AdaptiveInductiveClassifier, ::Type{<:EitherEnsembleModel{<:MLJFluxModel}}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Overloads the score function for ensembles of MLJFluxModel types.

function smooth_size_loss(
    conf_model::ConformalProbabilisticSet, fitresult, X;
    temp::Real=0.1, κ::Real=1.0
)

Computes the smooth (differentiable) size loss following Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. First, soft assignment probabilities are computed for new data X. Then (following the notation in the paper) the loss is computed as,

\[\Omega(C_{\theta}(x;\tau)) = \max (0, \sum_k C_{\theta,k}(x;\tau) - \kappa)\]

where $\tau$ is just the quantile q̂ and $\kappa$ is the target set size (defaults to $1$). For empty sets, the loss is computed as $K - \kappa$, that is the maximum set size minus the target set size.

ConformalPrediction.score(conf_model::SimpleInductiveClassifier, ::Type{<:EitherEnsembleModel{<:MLJFluxModel}}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Overloads the score function for ensembles of MLJFluxModel types.

ConformalPrediction.score(conf_model::InductiveModel, model::MLJFluxModel, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Overloads the score function for the MLJFluxModel type.

shape(model::NeuralNetworkRegressor, X, y)

A private method that returns the shape of the input and output of the model for given data X and y.

MLJFlux.train!(model::ConformalNN, penalty, chain, optimiser, X, y)

Implements the conformal traning procedure for the ConformalNN type.