### Abstract:

Many machine learning applications require
classifiers that minimize an asymmetric cost
function rather than the misclassification
rate, and several recent papers have addressed
this problem. However, these papers
have either applied no statistical testing
or have applied statistical methods that are
not appropriate for the cost-sensitive setting.
Without good statistical methods, it is difficult to tell whether these new cost-sensitive
methods are better than existing methods
that ignore costs, and it is also difficult to tell
whether one cost-sensitive method is better
than another. To rectify this problem, this
paper presents two statistical methods for the
cost-sensitive setting. The first constructs a
confidence interval for the expected cost of a
single classifier. The second constructs a confidence interval for the expected difference in
costs of two classifiers. In both cases, the
basic idea is to separate the problem of estimating
the probabilities of each cell in the
confusion matrix (which is independent of the
cost matrix) from the problem of computing
the expected cost. We show experimentally
that these bootstrap tests work better than
applying standard z tests based on the normal
distribution.