Data preparation and model training are important steps in the machine learning pipeline, but it’s also important to assess how well the learnt model is working. Depending on how well they generalise to new data, machine learning models can either be adaptive or non-adaptive.
By assessing our model’s performance according to a number of different criteria, we should be able to improve its overall predictive capability before we use it on fictitious data.
Puzzle Matrix
Presenting true positive (tp), true negative (TN), false positive (FP), and false negative (FN) predictions in this manner is frequent. These values are shown as a matrix, with the genuine classes on the Y axis and the expected classes on the X axis. Since it is based on class predictions, your model’s outputs must first be thresholded.
Error Type II
A type II error, commonly referred to as a false negative, takes place when a classification model predicts a wrong result for an initially valid observation. We increase the false negative rate when we fail to forecast something when it is imminent. Consider it as a portion of the fraudulent transactions that your model misses but allows to proceed.
Accuracy
The three measures that were just presented are all-purpose metrics, so you can use them regardless of the type of training and test data you have or the classification method you’ve used to solve your problems.
Precision
Classification accuracy is not always a reliable barometer of your model’s performance. When your class distribution is unbalanced, for example (one class is more frequent than others). This situation would result in a high accuracy rate even if you predicted that the most frequent class would apply to all data, which is completely illogical (because your model is not learning anything, and is just predicting everything as the top class).
Recall
Another crucial metric is recall, which is the percentage of samples from a class that the model correctly predicts. In a formal way:
Recall= True_Positive/ (True_Positive+ False_Negative)
Therefore, for our example above, the recall rate of cat and non-cat classes can be found as:
Recall_cat= 90/100= 90%
Recall_NonCat= 940/1000= 94%
