Downloading and visualizing model metrics

Alexander Lemm

Source: vignettes/model-metrics.Rmd

Zementis Server allows you to fetch model memory and model prediction metrics for each model that was deployed to the server. In this vignette we take a closer look at the function get_model_metrics() from zementisr that assists you in downloading the model metrics. Furthermore, we show you some best practices how to visualize the different model metrics of your predictive models.

Note: The features described in this vignette are only available for Zementis Server 10.3 or higher.

Some data science prep work

Before showing how to download the model metrics using the zementisr package, we will create two predictive models and upload them to the server: One classification model and one regression model. The prediction metrics returned from Zementis Server differ for these two model types.

We will use some shortcuts while following the data science process because we would like to concentrate on showcasing zementisr’s model metric-related functions in this vignette. For instance, we will just build the predictive models using the training sets without evaluating the model performances before deploying them to Zementis Server in PMML format. The test sets will be (mis-)used to simulate new and unseen data which is sent to Zementis Server for scoring.

Our classification model will be a classification tree created using the rpart() function and the iris data set. The classification tree will be used to predict the iris species.

We will use the built-in data set mtcars to create a multiple linear regression model for predicting car consumption measured in miles per gallon (MPG).

Now, we will convert both models to PMML and upload them to the server:

Retrieving model metrics from Zementis Server

Initially, get_model_metrics() will only return the memory related metrics expressed in MB if the model on the server has not yet predicted any new values which is the case for both models we just uploaded:

The memory metrics returned from get_model_metrics() are in fact a mix between server memory and model memory specific KPIs:

  • modelSize: The amount of memory consumed by the specific model itself
  • usedMemory: The entire memory currently used by all models and other resources on the server
  • freeMemory: The memory still available on the server
  • totalMemory: The total amount of server memory

Looking at the output above we see that both models only consume a tiny amount of memory. That makes sense because they represent really simple models which were trained using very small data sets. Ensemble models like a random forest model or a deep neural network trained with a lot of data including many predictor variables would consume far more memory.

Next, we will send 500 new values to each model deployed on Zementis Server for scoring. The data mimicking our new and unseen data will be samples we draw from the two test sets we created above.

After executing get_model_metrics() a second time, the response now includes the prediction-related metrics as well. As you can see the prediction metrics differ between classification and regression models.

For classification models the number of predictions for each class since model activation on the server are returned:

Prediction-related metrics for regression models include the Five-number summary since model activation on the server:

Visualizing model memory metrics

In this and in the next section we will use ggplot2 to visualize both the memory metrics and the prediction metrics for both models we just received from the server.

Let’s get started with visualizing the memory metrics:

Visualizing model prediction metrics

The prediction metrics are next. For classification models it makes most sense to create a bar chart based on the underlying data:

iris_metrics[["prediction_metrics"]] %>% 
  tidyr::gather("Class", "Number of cases") %>% 
  ggplot(aes(Class, `Number of cases`)) + 
    geom_col(aes(fill = Class)) + 
    coord_flip() + 
    labs(title = paste("Model prediction metrics for", iris_metrics[["model_name"]]),
         subtitle = paste("Predictions calcuated since model deployment:",
                          rowSums(iris_metrics[["prediction_metrics"]])))

The best visualization for the 5-number summary of the regression model is a boxplot:

Resetting the model metrics

Every time a model is deactivated on the server, the corresponding model metrics are automatically reset. Calling get_model_metrics() on a deactivated model just returns the model name. Neither the memory metrics nor the prediction metrics are returned in this case.

Let us demonstrate this behavior. First, we will deactivate the iris classification tree on the server:

Next, we try to download the model metrics again for the iris classification tree. However, this time the return list will only include the model name since the deactivation reset the complete model metrics:

Re-activating the model and calling get_model_metrics(), now returns the model name plus the memory model metrics:

The returned list above does not include the prediction metrics for the iris model since the model deactivation before completely reset these metrics. Not before new data for scoring is sent to the server, the prediction metrics will be included in the server’s response:

predict_pmml_batch(test_set_iris[1:5, ], "iris_model")