Technical Report
 

Bootstrap methods for the cost-sensitive evaluation of classifiers

Public Deposited

Downloadable Content

Download PDF
https://ir.library.oregonstate.edu/concern/technical_reports/zc77sr415

Descriptions

Attribute NameValues
Creator
Abstract
  • Many machine learning applications require classifiers that minimize an asymmetric cost function rather than the misclassification rate, and several recent papers have addressed this problem. However, these papers have either applied no statistical testing or have applied statistical methods that are not appropriate for the cost-sensitive setting. Without good statistical methods, it is difficult to tell whether these new cost-sensitive methods are better than existing methods that ignore costs, and it is also difficult to tell whether one cost-sensitive method is better than another. To rectify this problem, this paper presents two statistical methods for the cost-sensitive setting. The first constructs a confidence interval for the expected cost of a single classifier. The second constructs a confidence interval for the expected difference in costs of two classifiers. In both cases, the basic idea is to separate the problem of estimating the probabilities of each cell in the confusion matrix (which is independent of the cost matrix) from the problem of computing the expected cost. We show experimentally that these bootstrap tests work better than applying standard z tests based on the normal distribution.
Resource Type
Date Available
Date Issued
Series
Subject
Rights Statement
Funding Statement (additional comments about funding)
  • The authors gratefully acknowledge the support of NSF grants IRI-9626584 and EIA-9818414.
Publisher
Peer Reviewed
Language
Replaces

Relationships

Parents:

This work has no parents.

Items

Thumbnail Title Date Uploaded Visibility Actions
file details Bootstrap_methods_for_the_cost_sensitive_evaluation_of_classifiers.pdf 2017-07-18 Public Download