Problem Statement

Suppose you have three services, let’s call them hounds_service, breeding_service, and traits_service. Suppose further that breeding_service and traits_service both call my_hounds on hounds_service, which returns something including (from Theseus’ speech in A Midsummer Night’s Dream):

{
  breeding_stock: "Spartan",
  traits: {
    flewed_as: "Spartan",
    sanded_as: "Spartan"
  }
  …
}

For within-the-service tests, breeding_service and traits_service will stub out these responses. But they’ll also want tests in hounds_service that the values they care about appear in the actual response. If different teams own the different services, then, ideally, if the hounds_service team decides to rename breeding_stock to bred_from for their own reasons, they should see the broken test and know they need to go talk with the other team whose test on their code they just broke to come to some agreement about how or whether the field name needs to change consistently.

The simplest thing would be to add a comment:

# if this breaks, please talk to the breeding_service team
describe "my_hounds response includes 'breeding_stock'" do
  …

But it might be more efficient if there were some way of seeing at a glance which other tests (possibly on other repos) depend on this behaviour.

On the programming by wishful thinking approach, let’s introduce two new methods that we can put in spec blocks, requirement and relies_on. So:

# in hounds_service specs:

describe "my_hounds response includes 'breeding_stock'" do

  def requirement
    "my_hounds response includes 'breeding_stock'"
  end


# in breeding_service specs:

describe "code processing the my_hounds response" do

  def relies_on
    "hounds_service:my_hounds response includes 'breeding_stock'"
  end

Since the requirement is the same as the describe block’s description in this case, we might consider just using the description text, but starting with something that is more explicitly a search target, and less likely to be edited than a description, seems a safer first step.

Now we’ve got a concrete problem. Given a requirement, we want to locate and list all the relies_on blocks mentioning that requirement, either in the same repo or in other repos.

What we’ll want next is a relies_on lookup, where the keys are the relies_on text, and the values are the other spec example groups that have that text, and which we could interrogate as:

relied_on = relies_on_lookup.fetch(requirement, [])
if relied_on.any?
  # other specs have relies_on text matching our spec's requirement text
  # alert the user accordingly if our spec fails
end

(Edited to add: For a different approach, using comment annotations instead of method calls, see now also List Tests Relying on Another Test: Another Approach. But that very much builds on the work done here.)

In the Same Repo…

Let’s work with an example. Given a lib/service.rb:

class Service

  def my_hounds
    {
      breeding_stock: "Spartan",
      traits: {
        flewed_as: "Spartan",
        sanded_as: "Spartan"
      }
    }
  end
end

And a spec/with_requirement_spec.rb:

require "service"

RSpec.describe "with requirement" do

  subject { Service.new.my_hounds }

  context "my_hounds" do

    describe ":breeding_stock key" do
      def requirement
        "my_hounds includes :breeding_stock"
      end

      it "is included" do
        expect(subject).to include :breeding_stock
      end
    end

    describe ":traits key" do
      def requirement
        "my_hounds includes :traits"
      end

      it "is included" do
        expect(subject).to include :traits
      end
    end

    context "using my_hounds" do
      describe "for :breeding_stock" do
        def relies_on
          "my_hounds includes :breeding_stock"
        end

        it "does something with :breeding_stock" do
          expect(subject[:breeding_stock]).to eq("Spartan")
        end
      end
      describe "for :traits" do
        def relies_on
          "my_hounds includes :traits"
        end

        it "does something with :traits" do
          expect(subject[:traits][:flewed_as]).to eq("Spartan")
        end
      end
    end

    describe "using my_hounds :breeding_stock again" do
      def relies_on
        "my_hounds includes :breeding_stock"
      end

      it "checks something else" do
        expect(subject[:breeding_stock]).not_to eq("Athenian")
      end
    end
  end
end

That passes:

$ rspec
.....

Finished in 0.02194 seconds (files took 0.31486 seconds to load)
5 examples, 0 failures

And can be made to fail by commenting out line 5 of service:

$ rspec
F.F..

Failures:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include :breeding_stock

       expected {:traits => {:flewed_as => "Spartan", :sanded_as => "Spartan"}} to include :breeding_stock
       Diff:
       @@ -1 +1 @@
       -[:breeding_stock]
       +:traits => {:flewed_as=>"Spartan", :sanded_as=>"Spartan"},

     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

  etc… 

And we’d like the first failure message to include “Check out these other specs that rely on the assumption made in this one: …”.

Step 1: Build the relies_on lookup

Any spec file (or example group in a spec file) is going to be turned into a class extending RSpec::Core::ExampleGroup. So we can, at the beginning of the spec run, read ObjectSpace and find all the classes extending that:

  ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
    # do something with it
  end

Note that we need to do this at the beginning of the test run because ObjectSpace contains living objects and partway through the test run some of them might already be garbage collected and no longer there. (Yes, I have experimentally verified this.) We can put it into before(:suite), run once before everything else. We can’t save it to an instance variable from there, but we can add a setting to RSpec.configuration:

  config.add_setting :relies_on

  config.before(:suite) do
    config.relies_on = reckon_relies_on
  end

which we can access later as RSpec.configuration.relies_on.

In reckon_relies_on, we need to grab all the RSpec classes, instantiate them, see if they contain / respond to relies_on, and add them to the lookup if they do (note that multiple example groups might rely on the same requirement, like specs 3 and 5 above, so the value is an array)

def reckon_relies_on
  relies_on = Hash.new { |hash, key| hash[key] = [] }
  ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
    instance = klass.new
    next unless instance.respond_to?(:relies_on)

    relies_on_key = instance.method(:relies_on).call
    relies_on[relies_on_key] << klass
  end
  relies_on
end

If we make those changes in spec/spec_helper.rb we can put a binding.pry in the first spec, and in the console see what RSpec.configuration.relies_on is set to:

{
  "my_hounds includes :breeding_stock" =>
    [RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain,
     RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock],
  "my_hounds includes :traits" =>
    [RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForTraits]
}

At which point we have both the requirement (because we’re in the spec) and the relies_on lookup to check it against. If the spec passes, there’s nothing to do. But if it fails, and other specs rely on the requirement, we want to display that.

Step 2: Display the relied_on related specs in the failure output

We can start by building the addition to the error message that we wish we had. Add another method to spec/spec_helper.rb:

def relies_on_message(requirement)
  relies_on = RSpec.configuration.relies_on.fetch(requirement, [])
  if relies_on.any?
    "\nOther specs relying on this: \n- #{relies_on.join("\n- ")}"
  else
    ""
  end
end

And now, from the binding.pry in the first spec, if we call relies_on_message(requirement), we get:

Other specs relying on this: 
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock

To incorporate that into the actual error message, let’s build a custom matcher, which lets us customize the failure message. Add this as spec/support/matchers.rb (also add require 'support/matchers' to the top of spec/spec_helper.rb so that it can be used):

RSpec::Matchers.define :include_key do |key, requirement|
  match { |hash| hash.include?(key) }

  failure_message do
    "expected #{actual.keys} to include key #{key}, but it didn't\n- #{actual}\n#{relies_on_message(requirement)}"
  end

  failure_message_when_negated do
    "expected #{actual.keys} not to include key #{key}, but it did\n- #{actual}\n#{relies_on_message(requirement)}"
  end
end

And change the checks in the first two specs (lines 15 and 25) from (for line 15):

expect(subject).to include :breeding_stock

to

expect(subject).to include_key(:breeding_stock, requirement)

And then, if we comment out line 5 of lib/service.rb to break the first spec, we get the error message including the desired “Other specs relying on this assumption” block:

rspec
F.F..

Failures:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

   etc…

It is, granted, less than intuitive that we need to pass the requirement into the matcher – include_key(:breeding_stock, requirement) – what does requirement have to do with whether :breeding_stock is in my_hounds is a valid question, and the answer is nothing, save that we need requirement in there to check the relies_on lookup.

It is also the case that we’ll need to build other custom matchers for other kinds of checks if we want to include the relies_on_message(requirement) for those too.

But this still feels like a step in a good direction. Example repo with all changes described so far here.

A Rejected Alternate Approach

This doesn’t require custom matchers but does require a rescue for each spec in a block with a requirement, which felt worse, but, for completeness:

Add this to spec/spec_helper.rb:

def append_to_failure_message(e, requirement)
  e.message << relies_on_message(requirement)
  raise e
end

And in the spec, change line 15 from

  expect(subject).to include :breeding_stock

to

  expect(subject).to include(:breeding_stock)
rescue RSpec::Expectations::ExpectationNotMetError => e
  append_to_failure_message(e, requirement)

This does work, but using the rescue like this – even granted that it’s only going to happen when there’s a failing spec, so arguably it is a properly exceptional case instead of just control flow – didn’t sit well with me, hence the custom matcher approach I went with instead.

In Other Repos…

The simplest way to drive out the building of the other repos functionality is to create another repo with a spec with a relies_on method referring back to a requirement in this one:

def relies_on
  "relies_on_example:my_hounds includes :breeding_stock"
end

Then, in our repo, we need a list of the related_repos we need to search, a reference to what the related repos are calling our repo, and a way of cloning the related repos into our repo so we can examine them. We can add this to spec/spec_helper.rb:

def clone_related_repos
  FileUtils.rm_r("related_repos") if Dir.exist?("related_repos")
  FileUtils.mkdir("related_repos")
  FileUtils.cd("related_repos")
  related_repos.each do |repo|
    system("git clone #{repo}")
  end
end

def this_repo
  "relies_on_example"
end

def related_repos
  ["https://github.com/smiller/relies_on_example_related_repo"]
end

We’ll also want to add related_repos/ to .gitignore so we aren’t committing them.

Now if we call clone_related_repos in reckon_relies_on, we’ll have added a related_repos/ directory with the source code for all the related repos. We aren’t going to set them all running and examine their ObjectSpace (in the real not-simplified-for-a-blog-post example at the back of this there are a dozen related repos), so we need something simpler and quicker.

At this point, I put a binding.pry in reckon_relies_on and started experimenting. Using open3 to capture the results of grepping for "relies_on_example:

  stdout, _, _ = Open3.capture3("grep", "-nr", "\"#{this_repo}:", ".")

reveals

"./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4:      \"relies_on_example:my_hounds includes :breeding_stock\"\n"

(In a more complex project, this check might not be distinct enough to avoid false positives: in that case, we would want to make the related repo relies_on string more unique, for instance, <from:relies_on_example>:my_hounds include :breeding_stock), and search for that instead.)

From that, after some trial and error, we can further process the results and add them to the relies_on hash:

  matches = stdout.split("\n")
  matches.each do |line|
    file, key = line.split(/\:\s+\"/)
    key.sub!("#{this_repo}:", "")
      .sub!("\"", "")
    if file.include?("/spec/") && file.include?("_spec.rb:")
      relies_on[key] << file
    end
  end

And then, if we comment out line 5 of lib/service.rb and make the first spec fail again, we get a failure message including the line number of the relying-on spec in another repo:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
       - ./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4
     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

Further refinements are possible: if a github link to the line number in the other repo would be more likely to be followed up than a line number, we could change the code to provide that instead:

    if file.include?("/spec/") && file.include?("_spec.rb:")
      matches = /.\/(\w+)\/([\w\/.]+):(\d+)/.match(file)
      repo_url = related_repos.select { |x| x[x.rindex('/') + 1..] == matches[1] }.first
      repo_link = "#{repo_url}/blob/main/#{matches[2]}#L#{matches[3]}"
      relies_on[key] << repo_link
    end

resulting in:

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
       - https://github.com/smiller/relies_on_example_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L4

Changes to the repo from this section are here.

At the end of the day, to go back to the original example of the developer from one team wanting to change a key from :breeding_stock to :bred_from, this doesn’t stop them from doing that. But it does make much more clear the other parts of the codebase (even from different repos) that rely on the assumption staying the same, and should prompt conversations before the change is made.

Coda: After Sleeping on It…

I greatly prefer the convenience of the line number / github link references in the other repos code to the ObjectSpace references in the same repo code. Everything described here works, but I wonder if it would be even better to start from greppable comment annotations instead of methods, like so:

# requirement anchor
# @REQUIREMENT: my_hounds includes key :breeding_stock

# relies_on anchor in same repo
# @RELIES_ON: my_hounds includes key :breeding_stock

# relies_on anchor in another repo
# @RELIES_ON: <repo:hounds_service>:my_hounds includes key :breeding_stock

See now List Tests Relying on Another Test: Another Approach, where I’ve developed this version as well.

Appendix: The Thread from November 2019

This idea started in a twitter thread I wrote in November 2019, and recently re-read, and decided it was still a good idea:

[Germ of a testing idea] Suppose you’ve got a test. It depends on something elsewhere in the system behaving in a certain way. Within the test, you stub out that behaviour. Which is fine, provided your assumption holds.

If the other behaviour changes without warning, your assumption and your test become worse than useless. To prevent that, you make sure there’s a test elsewhere in the system testing the behaviour you’re relying on, and you add it if it isn’t already there.

But, if that behaviour changes – not just the interface, which we can auto-check, but the behaviour – it would be useful if there were an automatic way of notifying “Now you need to check these 12 places to see if they still work because they assumed the previous behaviour”

And so, maybe in addition to the tests-in-two-places, we need a marker on the dependent test, a depends-on annotation, some sort of mechanism that if you change behaviour X, it can report or “potentially or secondarily fail” all the places Y that depend on X.

This isn’t going back to the brittle tests problem of “you make a change and 27 tests start breaking”, or at least it’s different in that it tells you the one place to look at first.

It probably also indicates, if 26 other places depend on the behaviour, it should be treated as a public interface and not changed or only changed very carefully. So probably, in addition to running tests, we’d want to be able to build reports from it.

This is indirectly down to @alex_schl, as I was watching her keynote on exploratory testing from https://twitter.com/alex_schl/status/1196804674373980166 while thinking about how to improve some tests I’d written yesterday. All flaws are of course my own, and I will try to write this up with more detail soon.