Does Choice of Mutation Tool Matter? Public Deposited

http://ir.library.oregonstate.edu/concern/technical_reports/mw22v679m

Descriptions

Attribute NameValues
Creator
Abstract or Summary
  • Though mutation analysis is the primary means of evaluating the quality of test suites, though it suffers from inadequate standardization. Mutation analysis tools vary based on language, when mutants are generated (phase of compilation), and target audience. Mutation tools rarely implement the complete set of operators proposed in the literature, and most implement at least a few domain-specific mutation operators. Thus different tools may not always agree on the mutant kills of a test suite, and few criteria exist to guide a practitioner in choosing a tool, or a researcher in comparing previous results. We investigate an ensemble of measures such as traditional difficulty of detection, strength of minimal sets, diversity of mutants, as well as the information carried by the mutants produced, to evaluate the efficacy of mutant sets. By these measures, mutation tools rarely agree, often with large differences, and the variation due to project, even after accounting for difference due to test suites, is significant. However, the mean difference between tools is very small indicating that no single tool consistently skews mutation scores high or low for all projects. These results suggest that research using a single tool, a small number of projects, or small increments in mutation score may not yield reliable results. There is a clear need for greater standardization of mutation analysis; we propose one approach for such a standardization.
Resource Type
Date Available
Date Issued
Series
Rights Statement
Publisher
Peer Reviewed
Language
Replaces
Additional Information
  • description.provenance : Made available in DSpace on 2015-10-22T23:48:49Z (GMT). No. of bitstreams: 1 ieee_conf.pdf: 561252 bytes, checksum: 20d884aff04b47703b1e0af1cb801973 (MD5)
  • description.provenance : Submitted by Margaret Mellinger (margaret.mellinger@oregonstate.edu) on 2015-10-22T23:45:17Z No. of bitstreams: 1 ieee_conf.pdf: 561252 bytes, checksum: 20d884aff04b47703b1e0af1cb801973 (MD5)
  • description.provenance : Approved for entry into archive by Margaret Mellinger(margaret.mellinger@oregonstate.edu) on 2015-10-22T23:48:49Z (GMT) No. of bitstreams: 1 ieee_conf.pdf: 561252 bytes, checksum: 20d884aff04b47703b1e0af1cb801973 (MD5)

Relationships

Parents:

This work has no parents.

Last modified

Downloadable Content

Download PDF

Items