tidymodels: loss_accuracy provides no variable importance results

Question

Using the iris dataset, a knn-classifier was tuned with iterative search for multiple classification. However, using loss accuracy in DALEX::model_parts() for variable importance, provides empty results.

I would appreciate any ideas. Thank you so much for your support!

library(tidyverse)
library(tidymodels)
library(DALEXtra)
tidymodels_prefer()

df <- iris 

# split
set.seed(2023)
splits <- initial_split(df, strata = Species, prop = 4/5)
df_train <- training(splits)
df_test  <-  testing(splits)

# workflow
df_rec <- recipe(Species ~ ., data = df_train) 

knn_model <- nearest_neighbor(neighbors = tune()) %>% 
  set_engine("kknn") %>% 
  set_mode("classification")

df_wflow <- workflow() %>%
  add_model(knn_model) %>%
  add_recipe(df_rec) 

# cross-validation
set.seed(2023)
knn_res <-
  df_wflow %>%
  tune_bayes(
    metrics = metric_set(accuracy),
    resamples = vfold_cv(df_train, strata = "Species", v = 2),
    control = control_bayes(verbose = TRUE, save_pred = TRUE))

# fit
best_k <- knn_res %>%
  select_best("accuracy")

knn_mod <- df_wflow %>%
  finalize_workflow(best_k) %>%
  fit(df_train)

# variable importance
knn_exp <- explain_tidymodels(extract_fit_parsnip(knn_mod), 
                   data = df_rec %>% prep() %>% bake(new_data = NULL, all_predictors()),
                   y = df_train$Species)

set.seed(2023)
vip <- model_parts(knn_exp, type = "variable_importance", loss_function = loss_accuracy)
plot(vip) # empty plot

Answer 1

You are getting 0 for all your results because the the model type according to {DALEX} is "multiclass".

These calculations would have worked well if the type is "classification".

knn_exp$model_info$type
#> [1] "multiclass"

This means that the prediction that happens will be the predicted probabilities (here we get 1s and 0s because the modeling is quite overfit)

predicted <- knn_exp$predict_function(knn_exp$model, newdata = df_train)
predicted
#>      setosa versicolor virginica
#> [1,]      1          0         0
#> [2,]      1          0         0
#> [3,]      1          0         0
#> [4,]      1          0         0
#> [5,]      1          0         0
#> [6,]      1          0         0
#> ...

When you use loss_accuracy() as your loss function, it does that by using the following calculations

loss_accuracy
#> function (observed, predicted, na.rm = TRUE) 
#> mean(observed == predicted, na.rm = na.rm)
#> <bytecode: 0x159276bb8>
#> <environment: namespace:DALEX>
#> attr(,"loss_name")
#> [1] "Accuracy"

And we can see why this becomes an issue if we do the calculations steps by step. First we define the observed as the outcome factor

observed <- df_train$Species
observed
#>   [1] setosa     setosa     setosa     setosa     setosa     setosa    
#>   [7] setosa     setosa     setosa     setosa     setosa     setosa    
#>  [13] setosa     setosa     setosa     setosa     setosa     setosa    
#>  [19] setosa     setosa     setosa     setosa     setosa     setosa    
#>  [25] setosa     setosa     setosa     setosa     setosa     setosa    
#>  [31] setosa     setosa     setosa     setosa     setosa     setosa    
#>  [37] setosa     setosa     setosa     setosa     versicolor versicolor
#>  [43] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [49] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [55] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [61] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [67] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [73] versicolor versicolor versicolor versicolor versicolor versicolor
#>  [79] versicolor versicolor virginica  virginica  virginica  virginica 
#>  [85] virginica  virginica  virginica  virginica  virginica  virginica 
#>  [91] virginica  virginica  virginica  virginica  virginica  virginica 
#>  [97] virginica  virginica  virginica  virginica  virginica  virginica 
#> [103] virginica  virginica  virginica  virginica  virginica  virginica 
#> [109] virginica  virginica  virginica  virginica  virginica  virginica 
#> [115] virginica  virginica  virginica  virginica  virginica  virginica 
#> Levels: setosa versicolor virginica

since observed is a factor vector, and predicted is a numeric matrix we get back a logical matrix of FALSE since the values are never the same.

head(observed == predicted)
#>      setosa versicolor virginica
#> [1,]  FALSE      FALSE     FALSE
#> [2,]  FALSE      FALSE     FALSE
#> [3,]  FALSE      FALSE     FALSE
#> [4,]  FALSE      FALSE     FALSE
#> [5,]  FALSE      FALSE     FALSE
#> [6,]  FALSE      FALSE     FALSE

So when we take the mean of this we get the expected 0.

mean(observed == predicted)
#> [1] 0