Mutation Testing with Rspec

Question

I'm attempting to perform a mutation-test on some ruby code using rspec. I am just learning ruby and I don't really know if I'm doing it right. Part of the code I'm trying to test is:

class Cipher

 def initialize()
  @a_offset = 65 #'A'.unpack('C').first
  @z_offset = 90 #'Z'.unpack('C').first
 end

 def encode(key, plain_text)
  key_offset = key.upcase.unpack('C').first

  cipher_text = plain_text.upcase.split('').collect do |letter| 
   cipher_letter = (letter.unpack('C').first + key_offset - @a_offset)
   if cipher_letter > @z_offset
    cipher_letter -= ( @z_offset - @a_offset + 1 )
   end
   cipher_letter.chr
 end

 return cipher_text.join
end

So far my test suite looks like this:

require 'rspec'
require 'Cipher'

describe "#initialize" do 
   it "should have correct @a_offset" do
     encoder = Cipher.new()
     expect(encoder.instance_variable_get(:@a_offset)).to eq 65
   end 

   it "should have correct @z_offset" do
     encoder = Cipher.new()
     expect(encoder.instance_variable_get(:@z_offset)).to eq 90
   end
 end 

describe "#encode" do
   it "It should correctly encode Caesar with key = A"do
     encoder = Cipher.new()
     expect(encoder.encode('R', 'CAESAR')).to eq ("TRVJRI")
   end 
end

When running rspec my 3 tests pass. However when I use mutation testing on this suite I only kill 3/343 which is not very good.

Answer 1

Since your goal is to develop a mutation-adequate test suite, the unkilled mutants essentially provide guidance for what to test for. Ideally, your question should have provided some examples for generated mutants that are not killed by your test suite so that others can comment on how to write tests that kill them.

Let me try to best answer your question with general recommendations and examples. I am not familiar with the particular mutation testing tool you are using, but in general you should follow a process similar to the following*:

1. Select a mutant that is not killed by your test suite and determine whether it can be killed (a mutant might be semantically equivalent to your program -- if so, discard it from the pool of mutants).

2. Write a test that kills the selected mutant -- that is, write a test that passes on your original program but fails on the mutant.

   • Determine suitable inputs that trigger the faulty behavior, introduced by the mutant.
   • Determine a suitable test oracle that asserts on the expected behavior/output of your program (i.e., a test oracle that passes on your original program but fails on the mutant).

Please keep in mind that your goal should not be to write minimal tests that simply kill all mutants, but rather tests that make sense in the context of your project and as a byproduct kill the mutants. Again, if you could provide some concrete examples for mutants that your test suite does not kill, I and others can better comment on what tests are missing.

Example: Suppose you have a mutant that introduces the following fault (i.e., changes the relational operator in the if statement of your original program): - if cipher_letter > @z_offset + if cipher_letter >= @z_offset Killing this mutant requires a test that checks for the boundary condition -- that is, a test that encodes at least one character to a 'Z'. Your current test suite does not have such a test.

*This process describes traditional mutation testing. Recent research [1], [2] suggests that (1) not all killable mutants should be killed (some mutants elicit tests that simply don't make sense) and (2) equivalent mutants may indicate a problem in your program (e.g., unwanted redundancy or ambiguity) that you might want to fix instead of discarding the equivalent mutant.