Accuracy

An accuracy score is an evaluation out of 100%. It is used often in scientific experiment, however, it also applies to machine learning classification applications. It is calculated using equation:

$$ Accuracy=Correct\:Predictions/Total \:Data $$

The goal of a model is to achieve an accuracy of 100%, however, that rarely happens with validation dataset. Validation dataset is data that the model has never seen before.

An accuracy higher than 90% is considered as great for a classification application. With that said, an accuracy of 80%, or even 99%, for an imbalanced dataset is considered not good.

ROC-AUC

ROC-AUC score is a better measurement of the success of a classification algorithm because it does not take into account the imbalance of ratio of data. This score takes into account true positives, false positives, true negatives, and false negatives (see link for more info on what the terms mean).

Untitled

If the ROC-AUC score is illustrated, it should present a graph as above. The closer the blue line is to the top left corner, (0, 100), the better the discriminator is. In this case, we are using the term discriminator because the simple task of a classification model is to discriminate between two options, 0 or 1, fire or no fire.

A ROC-AUC score higher than 85% indicates a excellent classifier/discriminator, a score between 75%-85% indicates a moderate model, and one less than 0.75% indicates a bad model. In addition, if score is 50%, it indicates that the model cannot discriminate between the two options. A 50% or less score means the model has not learned anything.

Sources

Failure of Classification Accuracy for Imbalanced Class Distributions - Machine Learning Mastery

How to Use ROC Curves and Precision-Recall Curves for Classification in Python - Machine Learning Mastery