You Are the Only Possible Oracle: Effective Test Selection for End Users of Interactive Machine Learning Systems Public Deposited

http://ir.library.oregonstate.edu/concern/articles/dv13zv93n

©2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works. This is the author's peer-reviewed final manuscript, as accepted by the publisher. The published article is copyrighted by the Institute of Electrical and Electronics Engineers and can be found at:  http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=32.

Descriptions

Attribute NameValues
Creator
Abstract or Summary
  • How do you test a program when only a single user, with no expertise in software testing, is able to determine if the program is performing correctly? Such programs are common today in the form of machine-learned classifiers. We consider the problem of testing this common kind of machine-generated program when the only oracle is an end user: e.g., only you can determine if your email is properly filed. We present test selection methods that provide very good failure rates even for small test suites, and show that these methods work in both large-scale random experiments using a “gold standard” and in studies with real users. Our methods are inexpensive and largely algorithm-independent. Key to our methods is an exploitation of properties of classifiers that is not possible in traditional software testing. Our results suggest that it is plausible for time-pressured end users to interactively detect failures—even very hard-to-find failures—without wading through a large number of successful (and thus less useful) tests. We additionally show that some methods are able to find the arguably most difficult-to-detect faults of classifiers: cases where machine learning algorithms have high confidence in an incorrect result.
Resource Type
DOI
Date Available
Date Issued
Citation
  • Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett, M., Wong, W., ... & McIntosh, K. (2014). You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Transactions on Software Engineering, 40(3), 307-323. doi:10.1109/TSE.2013.59
Series
Keyword
Rights Statement
Funding Statement (additional comments about funding)
Publisher
Peer Reviewed
Language
Replaces
Additional Information
  • description.provenance : Submitted by Erin Clark (erin.clark@oregonstate.edu) on 2014-07-09T16:11:34Z No. of bitstreams: 1 GroceAlexElectEngCompSciYouAreOnlyPossible.pdf: 11680821 bytes, checksum: 0ea7c4468690d17f998fb3be9a8b3996 (MD5)
  • description.provenance : Approved for entry into archive by Erin Clark(erin.clark@oregonstate.edu) on 2014-07-09T16:11:50Z (GMT) No. of bitstreams: 1 GroceAlexElectEngCompSciYouAreOnlyPossible.pdf: 11680821 bytes, checksum: 0ea7c4468690d17f998fb3be9a8b3996 (MD5)
  • description.provenance : Made available in DSpace on 2014-07-09T16:11:50Z (GMT). No. of bitstreams: 1 GroceAlexElectEngCompSciYouAreOnlyPossible.pdf: 11680821 bytes, checksum: 0ea7c4468690d17f998fb3be9a8b3996 (MD5) Previous issue date: 2014-03

Relationships

In Administrative Set:
Last modified: 07/26/2017

Downloadable Content

Download PDF
Citations:

EndNote | Zotero | Mendeley

Items