Are there any good ruby testing traceability solutions?

Question

I'm writing some ruby (not Rails) and using test/unit with shoulda to write tests.

Are there any gems that'll allow me to implement traceability from my tests back to designs/requirements?

i.e.: I want to tag my tests with the name of the requirements they test, and then generate reports of requirements that aren't tested or have failing tests, etc.

Hopefully that's not too enterprisey for ruby.

Thanks!

Answer 1

Update: This solution is available as a gem: http://rubygems.org/gems/test4requirements

Are there any gems that'll allow me to implement traceability from my tests back to designs/requirements?

I don't know any gem, but your need was inspiration for a little experiment, how it could be solved.

• You have to define your Requirements with RequirementList.new(1,2,3,4)
• This requirements can be assigned with requirements in a TestCase
• each test can be assigned to a requirement with requirement
• after the test results you get an overview which requirements are tested (successfull)

And now the example:

gem 'test-unit'
require 'test/unit'

###########
# This should be a gem
###########   

class Test::Unit::TestCase
  def self.requirements(req)
    @@requirements = req
  end
  def requirement(req)
    raise RuntimeError, "No requirements defined for #{self}" unless defined? @@requirements
    caller.first =~ /:\d+:in `(.*)'/
    @@requirements.add_test(req, "#{self.class}##{$1}")
  end
  alias  :run_test_old :run_test
  def run_test
    run_test_old
    #this code is left if a problem occured.
    #in other words: if we reach this place, then the test was sucesfull
    if defined? @@requirements
      @@requirements.test_successfull("#{self.class}##{@method_name}")
    end
  end
end

class RequirementList
  def initialize( *reqs )
    @requirements = reqs
    @tests = {}
    @success = {}

    #Yes, we need two at_exit.
    #tests are done also at_exit.  With double at_exit, we are after that.
    #Maybe better to be added later.
    at_exit {
      at_exit do 
        self.overview
      end
    }

  end
  def add_test(key, loc)
    #fixme check duplicates
    @tests[key] = loc
  end
  def test_successfull(loc)
    #fixme check duplicates
    @success[loc] = true
  end
  def overview()
    puts "Requirements overiew"
    @requirements.each{|req|
      if @tests[req] #test defined
        if @success[@tests[req]]
          puts "Requirement #{req} was tested in #{@tests[req] }"
        else
          puts "Requirement #{req} was unsuccessfull tested in #{@tests[req] }"
        end
      else
        puts "Requirement #{req} was not tested"
      end
    }
  end
end #RequirementList

###############
## Here the gem end. The test will come.
###############

$req = RequirementList.new(1,2,3, 4)

class MyTest < Test::Unit::TestCase
  #Following requirements exist, and must be tested sucessfull
  requirements $req

  def test_1()
    requirement(1)  #this test is testing requirement 1
    assert_equal(2,1+1)
  end
  def test_2()
    requirement(2)
    assert_equal(3,1+1)
  end
  def test_3()
    #no assignment to requirement 3
    pend 'pend'
  end
end


class MyTest_4 < Test::Unit::TestCase
  #Following requirements exist, and must be tested sucessfull
  requirements $req

  def test_4()
    requirement(4)  #this test is testing requirement 4
    assert_equal(2,1+1)
  end
end

the result:

Loaded suite testing_traceability_solutions
Started
.FP.

  1) Failure:
test_2(MyTest)
    [testing_traceability_solutions.rb:89:in `test_2'
     testing_traceability_solutions.rb:24:in `run_test']:
<3> expected but was
<2>.

  2) Pending: pend
test_3(MyTest)
testing_traceability_solutions.rb:92:in `test_3'
testing_traceability_solutions.rb:24:in `run_test'

Finished in 0.65625 seconds.

4 tests, 3 assertions, 1 failures, 0 errors, 1 pendings, 0 omissions, 0 notifications
50% passed
Requirements overview:
Requirement 1 was tested in MyTest#test_1
Requirement 2 was unsuccessfull tested in MyTest#test_2
Requirement 3 was not tested
Requirement 4 was tested in MyTest_4#test_4

If you think, this could be a solution for you, please give me a feedback. Then I will try to build a gem out of it.

---

Code example for usage with shoulda

#~ require 'test4requirements' ###does not exist/use code above
require 'shoulda'
#use another interface ##not implemented###
#~ $req = Requirement.new_from_file('requirments.txt')

class MyTest_shoulda < Test::Unit::TestCase
  #Following requirements exist, and must be tested sucessfull
  #~ requirements $req

  context 'req. of customer X' do
    #Add requirement as parameter of should
    # does not work yet
    should 'fullfill request 1', requirement: 1  do
      assert_equal(2,1+1)
    end
    #add requirement via requirement command
    #works already
    should 'fullfill request 1' do
      requirement(1)  #this test is testing requirement 1
      assert_equal(2,1+1)
    end
  end #context
end    #MyTest_shoulda

Answer 2

With cucumber you can have your requirement be the test, doesn't get any more traceable than that :)

So a single requirement is a feature, and a feature has scenario that you want to test.

# addition.feature

Feature: Addition
  In order to avoid silly mistakes
  As a math idiot 
  I want to be told the sum of two numbers

  Scenario Outline: Add two numbers
    Given I have entered <input_1> into the calculator
    And I have entered <input_2> into the calculator
    When I press <button>
    Then the result should be <output> on the screen

  Examples:
    | input_1 | input_2 | button | output |
    | 20      | 30      | add    | 50     |
    | 2       | 5       | add    | 7      |
    | 0       | 40      | add    | 40     |

Then you have step definitions written in ruby mapped to the scenario

# step_definitons/calculator_steps.rb

begin require 'rspec/expectations'; rescue LoadError; require 'spec/expectations'; end 
require 'cucumber/formatter/unicode'
$:.unshift(File.dirname(__FILE__) + '/../../lib')
require 'calculator'

Before do
  @calc = Calculator.new
end

After do
end

Given /I have entered (\d+) into the calculator/ do |n|
  @calc.push n.to_i
end

When /I press (\w+)/ do |op|
  @result = @calc.send op
end

Then /the result should be (.*) on the screen/ do |result|
  @result.should == result.to_f
end