Posts Tagged → rspec

List Tests Relying on Another Test: Another Approach

Problem Statement

See the previous post. TL;DR: we want a mechanism that lets us link a test in one part of a codebase to other tests that rely on the assumption of that initial test, say that an API call response will include certain fields. If one team wants to make a change to that response, we want to make it clear what else might break if they make that change, and who they should probably talk with to build a shared understanding of what should be done next.

In the previous post we looked at doing this with def requirement and def relies_on methods: here we’re going to look instead at doing it with comment annotations, so:

# requirement anchor
# @REQUIREMENT: my_hounds includes key :breeding_stock

# relies_on anchor in same repo
# @RELIES_ON: my_hounds includes key :breeding_stock

# relies_on anchor in another repo
# @RELIES_ON: <repo:hounds_service>:my_hounds includes key :breeding_stock

Rewriting the Example

Let’s rewrite the example. lib/service.rb stays the same:

class Service

  def my_hounds
    {
      breeding_stock: "Spartan",
      traits: {
        flewed_as: "Spartan",
        sanded_as: "Spartan"
      }
    }
  end
end

The spec/with_requirement_spec.rb changes its requirement and relies_on from methods to comment annotations, and already this lets us put them on individual examples and not just within example groups, giving us more flexibility:

require "service"

RSpec.describe "with requirement" do

  subject { Service.new.my_hounds }

  context "my_hounds" do
    describe "keys in response" do
      # @REQUIREMENT: my_hounds includes :breeding_stock
      it "includes :breeding_stock" do
        expect(subject).to include :breeding_stock
      end

      # @REQUIREMENT: my_hounds includes :traits
      it "includes :traits" do
        expect(subject).to include :traits
      end
    end
  end

  context "using my_hounds" do
    describe "for one thing" do
      # @RELIES_ON: my_hounds includes :breeding_stock
      it "does something with :breeding_stock" do
        expect(subject[:breeding_stock]).to eq("Spartan")
      end

      # @RELIES_ON: my_hounds includes :traits
      it "does something with :traits" do
        expect(subject[:traits][:flewed_as]).to eq("Spartan")
      end
    end

    describe "for another thing" do
      # @RELIES_ON: my_hounds includes :breeding_stock
      it "checks something else" do
        expect(subject[:breeding_stock]).not_to eq("Athenian")
      end
    end
  end
end

As before, this passes, and can be made to fail easily by commenting out line 5 of the lib/service.rb.

We can also create a related repo, with a spec referring back to the requirement in the first repo:

RSpec.describe "relies on related repo" do
  describe "my_hounds from relies_on_example" do
    # @RELIES_ON: <repo:relies_on_example_two>:my_hounds includes :breeding_stock
    it "does something" do
      expect(true).to be true
    end
  end
end

Building the Relies_on Lookup

It still makes sense to do this in the before(:suite), and to store the results in the added setting config.relies_on. So add in spec/spec_helper.rb, as before:

config.add_setting :relies_on

config.before(:suite) do
  config.relies_on = reckon_relies_on
end

But instead of reading ObjectSpace, reckon_relies_on can start grepping instead. Running

def reckon_relies_on
  stdout, _, _ = Open3.capture3("grep", "-nr", "# @RELIES_ON", "spec")
  binding.pry
end

and looking at stdout gives us:

"spec/spec_helper.rb:110:  stdout, _, _ = Open3.capture3(\"grep\", \"-nr\", \"# @RELIES_ON\", \".\")\n
spec/with_requirement_spec.rb:23:      # @RELIES_ON: my_hounds includes :breeding_stock\n
spec/with_requirement_spec.rb:28:      # @RELIES_ON: my_hounds includes :traits\n
spec/with_requirement_spec.rb:35:      # @RELIES_ON: my_hounds includes :breeding_stock\n"

And by experimentation, and reusing / rewriting some of the grepping code from the last example, we can end up with:

def repos
  [
    # current repo
    {
      repo_url: "https://github.com/smiller/relies_on_example_two",
      source_directory: "./",
      relies_on_label: "# @RELIES_ON: "
    },
    # related repos
    {
      repo_url: "https://github.com/smiller/relies_on_example_two_related_repo",
      source_directory: "./relies_on_example_two_related_repo/",
      relies_on_label: "# @RELIES_ON: <repo:relies_on_example_two>:"
    }
  ]
end

def reckon_relies_on
  relies_on = Hash.new { |hash, key| hash[key] = [] }
  FileUtils.rm_r("related_repos") if Dir.exist?("related_repos")

  repo = repos[0]
  relies_on = reckon_relies_on_for_one_repo(repo[:repo_url], repo[:source_directory], repo[:relies_on_label], relies_on)

  FileUtils.mkdir("related_repos")
  FileUtils.cd("related_repos")
  repos[1..].each do |repo|
    system("git clone #{repo[:repo_url]}")
    relies_on = reckon_relies_on_for_one_repo(repo[:repo_url], repo[:source_directory], repo[:relies_on_label], relies_on)
  end
  relies_on
end

def reckon_relies_on_for_one_repo(repo_url, source_directory, relies_on_label, relies_on)
  stdout, _, _ = Open3.capture3("grep", "-nr", relies_on_label, "#{source_directory}spec")
  lines = stdout.split("\n")
  lines.each do |line|
    file, key = line.split(/\:\s+#{relies_on_label}/)
    if file.include?("_spec.rb:")
      matches = /#{source_directory}([\w\/\.]+):(\d+)/.match(file)
      repo_link = "#{repo_url}/blob/main/#{matches[1]}#L#{matches[2]}"
      relies_on[key] << repo_link
    end
  end
  relies_on
end

After which, RSpec.configuration.relies_on is set to:

{"my_hounds includes :breeding_stock" =>
   ["https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L23",
    "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L35",
    "https://github.com/smiller/relies_on_example_two_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L3"],
 "my_hounds includes :traits" =>
   ["https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L28"]}

… which, checking against the github links, is what we expect it to be. So the relies_on lookup is now done.

Finding the Requirement

If we comment out line 5 of lib/service.rb to make the specs fail, we see that while it was simpler to build the relies_on lookup in this version, it’s going to be harder to get the requirement programmatically. When it was a method it was in scope so we could just call it to get the result, but now that it’s a comment annotation, we’ll have to work it out.

The simplest solution is a cheat with a manual step, but we’ll put it in to see the rest of the code work. We introduced a custom matcher in the previous post that takes the requirement as the argument. Let’s reintroduce it here, as spec/support/matchers.rb (also require 'support/matchers' in spec/spec_helper.rb so that it is accessible):

RSpec::Matchers.define :include_key do |key, requirement|
  match { |hash| hash.include?(key) }

  failure_message do
    "expected #{actual.keys} to include key #{key}, but it didn't\n- #{actual}\n#{relies_on_message(requirement)}"
  end

  failure_message_when_negated do
    "expected #{actual.keys} not to include key #{key}, but it did\n- #{actual}\n#{relies_on_message(requirement)}"
  end
end

Since that uses the relies_on_message(requirement) method, let’s add that back into spec/spec_helper.rb too:

def relies_on_message(requirement)
  relies_on = RSpec.configuration.relies_on.fetch(requirement, [])
  if relies_on.any?
    "\nOther specs relying on this: \n- #{relies_on.join("\n- ")}"
  else
    ""
  end
end

Now, since we’re explicitly (and only) calling the new include_key matcher when we know we’ve got a requirement, and since we can see from the annotation above the method what the requirement is:

      # @REQUIREMENT: my_hounds includes :breeding_stock
      it "includes :breeding_stock" do
        expect(subject).to include :breeding_stock
      end

we could change line 11 of with_requirement_spec.rb from

        expect(subject).to include :breeding_stock

to 

        expect(subject).to include_key(:breeding_stock, "my_hounds includes :breeding_stock")

copying the requirement from line 9. This is a manual step and a duplication and we’d very much prefer to get rid of it, but if we try that now and run the specs, we do get the expected error message with related github links:

  1) with requirement my_hounds keys in response includes :breeding_stock
     Failure/Error: expect(subject).to include_key(:breeding_stock, "my_hounds includes :breeding_stock")

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L23"
       - "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L35"
       - "https://github.com/smiller/relies_on_example_two_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L3"
     # ./spec/with_requirement_spec.rb:11:in `block (4 levels) in <top (required)>'

(The quotation marks aren’t in the actual output: I’ve added them in because without them WordPress leaves off the line number suffix (e.g. #L23) from the link, so it just goes to the file, but with them it includes the line number suffix, so if you click on the links above within their quotation marks they go to the file and line number.)

Changes from this section are here.

Finding the Requirement without Cheating

It was good to see everything working, but it feels wrong to be manually copying a string when it’s two lines away. Can we do better? Let’s see.

If put back our binding.pry and inspect the example, we see it gives us the file and the line number where the example begins:

self.inspect
=> "#<RSpec::ExampleGroups::WithRequirement::MyHounds::KeysInResponse \"includes :breeding_stock\" (./spec/with_requirement_spec.rb:10)>"

We can regexp to get out the file and line number from that:

matches = /.+\((.+):(\d+)\).+/.match(self.inspect)
=> #<MatchData
 "#<RSpec::ExampleGroups::WithRequirement::MyHounds::KeysInResponse \"includes :breeding_stock\" (./spec/with_requirement_spec.rb:10)>"
 1:"./spec/with_requirement_spec.rb"
 2:"10">

This feels potentially fragile and accidental, but let’s run with it for now. Now that we’ve got the file, we can check for # @REQUIREMENT: comments within it:

stdout, _, _ = Open3.capture3("grep", "-nr", "# @REQUIREMENT: ", matches[1])
stdout
=> "./spec/with_requirement_spec.rb:9:      # @REQUIREMENT: my_hounds includes :breeding_stock\n./spec/with_requirement_spec.rb:15:      # @REQUIREMENT: my_hounds includes :traits\n"

And if the requirement is at the example level, we know its line_number will be matches[2].to_i - 1 => 9, so we can iterate over the requirement lines looking for that:

  lines = stdout.split("\n")
  lines.each do |line|
    line_matches = /#{example_matches[1]}:(\d+).+# @REQUIREMENT: (.+)/.match(line)
    if line_matches[1].to_i == line_number
      puts "REQUIREMENT IS: #{line_matches[2]}"
    end
  end

So we can create a reckon_requirement(example) method in spec/spec_helper.rb:

def reckon_requirement(example)
  example_matches = /.+\((.+):(\d+)\).+/.match(example.inspect)
  line_number = example_matches[2].to_i - 1
  stdout, _, _ = Open3.capture3("grep", "-nr", "# @REQUIREMENT: ", example_matches[1])
  lines = stdout.split("\n")
  lines.each do |line|
    line_matches = /#{example_matches[1]}:(\d+).+# @REQUIREMENT: (.+)/.match(line)
    if line_matches[1].to_i == line_number
      return line_matches[2]
    end
  end
end

And call it instead where we’re passing in the manually-typed requirement in line 11 of with_requirement_spec.rb:

        expect(subject).to include_key(:breeding_stock, reckon_requirement(self))

Then, in this case, because the comment annotation is on the specific example, it works, and we get the expected “Other specs relying on this: …” error message without manually copying the requirement:

  1) with requirement my_hounds keys in response includes :breeding_stock
     Failure/Error: expect(subject).to include_key(:breeding_stock, reckon_requirement(self))

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L23"
       - "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L35"
       - "https://github.com/smiller/relies_on_example_two_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L3"
     # ./spec/with_requirement_spec.rb:11:in `block (4 levels) in <top (required)>'

This does depend on the assumption that the requirement comment annotation will be at the example level, not on an example group level, which is not a restriction we had with the previous solution. But it’s probably a reasonable assumption. And if it ever isn’t, we can either dig deeper into a more complex automated solution or pass in a manually-copied string.

(But if we do find ourselves needing to pass in manually-copied strings, we should at least write a spec that gathers all the requirements from the comment annotations and all of the explicit requirement strings passed into custom matchers and verify that all the custom strings are valid requirements from the list of comment annotations.)

But if we’re good with the restriction that requirement comment annotation is at the specific example level, this now works automatically. Changes from this section are here.

Next-Day Improvement

Having just watched Noel Rappin’s RailsConf 2023 talk, I was reminded that Ruby gives us __FILE__ and __LINE__ to read the current file name and line number, which feels better and more direct than using a regexp as we did above.

Since reckon_requirement only needs the file name and line number, let’s change it from reckon_requirement(example) to reckon_requirement(file_name, line_number). In fact, we can pass in the line number of the requirement (in our case, __LINE__ - 2, but this also gives us the flexibility to reference a requirement on an example group and not just use the one on the example, which we didn’t have before), and, in case we make a mistake counting line numbers, let’s raise an error in reckon_requirement if it doesn’t find a requirement at the specified line number.

We change the call in the spec from:

      # @REQUIREMENT: my_hounds includes :breeding_stock
      it "includes :breeding_stock" do
        expect(subject).to include_key(:breeding_stock, reckon_requirement(self))
      end

to

      # @REQUIREMENT: my_hounds includes :breeding_stock
      it "includes :breeding_stock" do
        expect(subject).to include_key(:breeding_stock, reckon_requirement(__FILE__, __LINE__ - 2)) 
        # __LINE__ is 11, and the requirement is on line 9, hence __LINE__ - 2
      end

and we change reckon_requirement in spec_helper.rb to:

def reckon_requirement(full_path_file_name, line_number)
  file_name = "./#{full_path_file_name[`pwd`.length..]}"
  stdout, _, _ = Open3.capture3("grep", "-nr", "# @REQUIREMENT: ", file_name)
  lines = stdout.split("\n")
  lines.each do |line|
    matches = /#{file_name}:(\d+).+# @REQUIREMENT: (.+)/.match(line)
    if matches[1].to_i == line_number
      return matches[2]
    end
  end
  raise "requirement expected but not found at file #{file_name} line #{line_number}"
end

We can change the line_number we pass in at line 11 to __LINE__ - 3 (line 8, the line above the requirement) to verify we get the error with a usefully specific message:

Failures:

  1) with requirement my_hounds keys in response includes :breeding_stock
     Failure/Error: raise "requirement expected but not found at file #{file_name} line #{line_number}"

     RuntimeError:
       requirement expected but not found at file ./spec/with_requirement_spec.rb line 8
     # ./spec/spec_helper.rb:177:in `reckon_requirement'
     # ./spec/with_requirement_spec.rb:11:in `block (4 levels) in <top (required)>'

From requirement expected but not found at file ./spec/with_requirement_spec.rb line 8 we can look back at our spec file 

    describe "keys in response" do
      # @REQUIREMENT: my_hounds includes :breeding_stock
      it "includes :breeding_stock" do
        expect(subject).to include_key(:breeding_stock, reckon_requirement(__FILE__, __LINE__ - 2))
      end

and see we meant line 9 / __LINE__ - 2 (not line 8 / __LINE - 3), and fix it easily.

Having fixed that, we can also comment out the line in lib/service.rb and verify that we still get the error message including the other specs relying on this block.

Looking at that block:

Other specs relying on this:
- "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L23"
- "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L35"
- "https://github.com/smiller/relies_on_example_two_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L3"

It occurs that naming the requirement instead of just saying “this” would probably be more useful, so we change relies_on_message(requirement) to say

    "\nOther specs relying on requirement '#{requirement}': \n- #{relies_on.join("\n- ")}"

And re-run to get a more informative error:

Other specs relying on requirement 'my_hounds includes :breeding_stock':
- "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L23"
- "https://github.com/smiller/relies_on_example_two/blob/main/spec/with_requirement_spec.rb#L35"
- "https://github.com/smiller/relies_on_example_two_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L3"

That feels better. Changes from this section are here.

Initial Conclusion

I think I prefer this solution to the one from the previous post, largely because of the github-to-specific-line-number links in all cases. It is true that we’re encoding data in comments, which can be risky, but since they’re clearly structured comments – # @REQUIREMENT: and # @RELIES_ON: – they feel more secure.

Also, since we’ve now got worked examples of both, we can let them breathe side-by-side for a bit, try them out on other examples, possibly / probably discover different edge cases, and come to a better-founded conclusion. It’s a lot easier to reason about now that both exist as running code.

List Tests Relying on Another Test

Problem Statement

Suppose you have three services, let’s call them hounds_service, breeding_service, and traits_service. Suppose further that breeding_service and traits_service both call my_hounds on hounds_service, which returns something including (from Theseus’ speech in A Midsummer Night’s Dream):

{
  breeding_stock: "Spartan",
  traits: {
    flewed_as: "Spartan",
    sanded_as: "Spartan"
  }
  …
}

For within-the-service tests, breeding_service and traits_service will stub out these responses. But they’ll also want tests in hounds_service that the values they care about appear in the actual response. If different teams own the different services, then, ideally, if the hounds_service team decides to rename breeding_stock to bred_from for their own reasons, they should see the broken test and know they need to go talk with the other team whose test on their code they just broke to come to some agreement about how or whether the field name needs to change consistently.

The simplest thing would be to add a comment:

# if this breaks, please talk to the breeding_service team
describe "my_hounds response includes 'breeding_stock'" do
  …

But it might be more efficient if there were some way of seeing at a glance which other tests (possibly on other repos) depend on this behaviour.

On the programming by wishful thinking approach, let’s introduce two new methods that we can put in spec blocks, requirement and relies_on. So:

# in hounds_service specs:

describe "my_hounds response includes 'breeding_stock'" do

  def requirement
    "my_hounds response includes 'breeding_stock'"
  end


# in breeding_service specs:

describe "code processing the my_hounds response" do

  def relies_on
    "hounds_service:my_hounds response includes 'breeding_stock'"
  end

Since the requirement is the same as the describe block’s description in this case, we might consider just using the description text, but starting with something that is more explicitly a search target, and less likely to be edited than a description, seems a safer first step.

Now we’ve got a concrete problem. Given a requirement, we want to locate and list all the relies_on blocks mentioning that requirement, either in the same repo or in other repos.

What we’ll want next is a relies_on lookup, where the keys are the relies_on text, and the values are the other spec example groups that have that text, and which we could interrogate as:

relied_on = relies_on_lookup.fetch(requirement, [])
if relied_on.any?
  # other specs have relies_on text matching our spec's requirement text
  # alert the user accordingly if our spec fails
end

(Edited to add: For a different approach, using comment annotations instead of method calls, see now also List Tests Relying on Another Test: Another Approach. But that very much builds on the work done here.)

In the Same Repo…

Let’s work with an example. Given a lib/service.rb:

class Service

  def my_hounds
    {
      breeding_stock: "Spartan",
      traits: {
        flewed_as: "Spartan",
        sanded_as: "Spartan"
      }
    }
  end
end

And a spec/with_requirement_spec.rb:

require "service"

RSpec.describe "with requirement" do

  subject { Service.new.my_hounds }

  context "my_hounds" do

    describe ":breeding_stock key" do
      def requirement
        "my_hounds includes :breeding_stock"
      end

      it "is included" do
        expect(subject).to include :breeding_stock
      end
    end

    describe ":traits key" do
      def requirement
        "my_hounds includes :traits"
      end

      it "is included" do
        expect(subject).to include :traits
      end
    end

    context "using my_hounds" do
      describe "for :breeding_stock" do
        def relies_on
          "my_hounds includes :breeding_stock"
        end

        it "does something with :breeding_stock" do
          expect(subject[:breeding_stock]).to eq("Spartan")
        end
      end
      describe "for :traits" do
        def relies_on
          "my_hounds includes :traits"
        end

        it "does something with :traits" do
          expect(subject[:traits][:flewed_as]).to eq("Spartan")
        end
      end
    end

    describe "using my_hounds :breeding_stock again" do
      def relies_on
        "my_hounds includes :breeding_stock"
      end

      it "checks something else" do
        expect(subject[:breeding_stock]).not_to eq("Athenian")
      end
    end
  end
end

That passes:

$ rspec
.....

Finished in 0.02194 seconds (files took 0.31486 seconds to load)
5 examples, 0 failures

And can be made to fail by commenting out line 5 of service:

$ rspec
F.F..

Failures:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include :breeding_stock

       expected {:traits => {:flewed_as => "Spartan", :sanded_as => "Spartan"}} to include :breeding_stock
       Diff:
       @@ -1 +1 @@
       -[:breeding_stock]
       +:traits => {:flewed_as=>"Spartan", :sanded_as=>"Spartan"},

     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

  etc…

And we’d like the first failure message to include “Check out these other specs that rely on the assumption made in this one: …”.

Step 1: Build the relies_on lookup

Any spec file (or example group in a spec file) is going to be turned into a class extending RSpec::Core::ExampleGroup. So we can, at the beginning of the spec run, read ObjectSpace and find all the classes extending that:

  ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
    # do something with it
  end

Note that we need to do this at the beginning of the test run because ObjectSpace contains living objects and partway through the test run some of them might already be garbage collected and no longer there. (Yes, I have experimentally verified this.) We can put it into before(:suite), run once before everything else. We can’t save it to an instance variable from there, but we can add a setting to RSpec.configuration: 

  config.add_setting :relies_on

  config.before(:suite) do
    config.relies_on = reckon_relies_on
  end

which we can access later as RSpec.configuration.relies_on.

In reckon_relies_on, we need to grab all the RSpec classes, instantiate them, see if they contain / respond to relies_on, and add them to the lookup if they do (note that multiple example groups might rely on the same requirement, like specs 3 and 5 above, so the value is an array)

def reckon_relies_on
  relies_on = Hash.new { |hash, key| hash[key] = [] }
  ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
    instance = klass.new
    next unless instance.respond_to?(:relies_on)

    relies_on_key = instance.method(:relies_on).call
    relies_on[relies_on_key] << klass
  end
  relies_on
end

If we make those changes in spec/spec_helper.rb we can put a binding.pry in the first spec, and in the console see what RSpec.configuration.relies_on is set to: 

{
  "my_hounds includes :breeding_stock" =>
    [RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain,
     RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock],
  "my_hounds includes :traits" =>
    [RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForTraits]
}

At which point we have both the requirement (because we’re in the spec) and the relies_on lookup to check it against. If the spec passes, there’s nothing to do. But if it fails, and other specs rely on the requirement, we want to display that.

Step 2: Display the relied_on related specs in the failure output

We can start by building the addition to the error message that we wish we had. Add another method to spec/spec_helper.rb:

def relies_on_message(requirement)
  relies_on = RSpec.configuration.relies_on.fetch(requirement, [])
  if relies_on.any?
    "\nOther specs relying on this: \n- #{relies_on.join("\n- ")}"
  else
    ""
  end
end

And now, from the binding.pry in the first spec, if we call relies_on_message(requirement), we get:

Other specs relying on this: 
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock

To incorporate that into the actual error message, let’s build a custom matcher, which lets us customize the failure message. Add this as spec/support/matchers.rb (also add require 'support/matchers' to the top of spec/spec_helper.rb so that it can be used):

RSpec::Matchers.define :include_key do |key, requirement|
  match { |hash| hash.include?(key) }

  failure_message do
    "expected #{actual.keys} to include key #{key}, but it didn't\n- #{actual}\n#{relies_on_message(requirement)}"
  end

  failure_message_when_negated do
    "expected #{actual.keys} not to include key #{key}, but it did\n- #{actual}\n#{relies_on_message(requirement)}"
  end
end

And change the checks in the first two specs (lines 15 and 25) from (for line 15):

expect(subject).to include :breeding_stock

to

expect(subject).to include_key(:breeding_stock, requirement)

And then, if we comment out line 5 of lib/service.rb to break the first spec, we get the error message including the desired “Other specs relying on this assumption” block:

rspec
F.F..

Failures:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

   etc…

It is, granted, less than intuitive that we need to pass the requirement into the matcher – include_key(:breeding_stock, requirement) – what does requirement have to do with whether :breeding_stock is in my_hounds is a valid question, and the answer is nothing, save that we need requirement in there to check the relies_on lookup.

It is also the case that we’ll need to build other custom matchers for other kinds of checks if we want to include the relies_on_message(requirement) for those too.

But this still feels like a step in a good direction. Example repo with all changes described so far here.

A Rejected Alternate Approach

This doesn’t require custom matchers but does require a rescue for each spec in a block with a requirement, which felt worse, but, for completeness:

Add this to spec/spec_helper.rb:

def append_to_failure_message(e, requirement)
  e.message << relies_on_message(requirement)
  raise e
end

And in the spec, change line 15 from

  expect(subject).to include :breeding_stock

to

  expect(subject).to include(:breeding_stock)
rescue RSpec::Expectations::ExpectationNotMetError => e
  append_to_failure_message(e, requirement)

This does work, but using the rescue like this – even granted that it’s only going to happen when there’s a failing spec, so arguably it is a properly exceptional case instead of just control flow – didn’t sit well with me, hence the custom matcher approach I went with instead.

In Other Repos…

The simplest way to drive out the building of the other repos functionality is to create another repo with a spec with a relies_on method referring back to a requirement in this one:

def relies_on
  "relies_on_example:my_hounds includes :breeding_stock"
end

Then, in our repo, we need a list of the related_repos we need to search, a reference to what the related repos are calling our repo, and a way of cloning the related repos into our repo so we can examine them. We can add this to spec/spec_helper.rb:

def clone_related_repos
  FileUtils.rm_r("related_repos") if Dir.exist?("related_repos")
  FileUtils.mkdir("related_repos")
  FileUtils.cd("related_repos")
  related_repos.each do |repo|
    system("git clone #{repo}")
  end
end

def this_repo
  "relies_on_example"
end

def related_repos
  ["https://github.com/smiller/relies_on_example_related_repo"]
end

We’ll also want to add related_repos/ to .gitignore so we aren’t committing them.

Now if we call clone_related_repos in reckon_relies_on, we’ll have added a related_repos/ directory with the source code for all the related repos. We aren’t going to set them all running and examine their ObjectSpace (in the real not-simplified-for-a-blog-post example at the back of this there are a dozen related repos), so we need something simpler and quicker.

At this point, I put a binding.pry in reckon_relies_on and started experimenting. Using open3 to capture the results of grepping for "relies_on_example:

  stdout, _, _ = Open3.capture3("grep", "-nr", "\"#{this_repo}:", ".")

reveals 

"./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4:      \"relies_on_example:my_hounds includes :breeding_stock\"\n"

(In a more complex project, this check might not be distinct enough to avoid false positives: in that case, we would want to make the related repo relies_on string more unique, for instance, <from:relies_on_example>:my_hounds include :breeding_stock), and search for that instead.)

From that, after some trial and error, we can further process the results and add them to the relies_on hash:

  matches = stdout.split("\n")
  matches.each do |line|
    file, key = line.split(/\:\s+\"/)
    key.sub!("#{this_repo}:", "")
      .sub!("\"", "")
    if file.include?("/spec/") && file.include?("_spec.rb:")
      relies_on[key] << file
    end
  end

And then, if we comment out line 5 of lib/service.rb and make the first spec fail again, we get a failure message including the line number of the relying-on spec in another repo:

  1) with requirement my_hounds :breeding_stock key is included
     Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)

       expected [:traits] to include key breeding_stock, but it didn't
       - {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
       - ./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4
     # ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'

Further refinements are possible: if a github link to the line number in the other repo would be more likely to be followed up than a line number, we could change the code to provide that instead:

    if file.include?("/spec/") && file.include?("_spec.rb:")
      matches = /.\/(\w+)\/([\w\/.]+):(\d+)/.match(file)
      repo_url = related_repos.select { |x| x[x.rindex('/') + 1..] == matches[1] }.first
      repo_link = "#{repo_url}/blob/main/#{matches[2]}#L#{matches[3]}"
      relies_on[key] << repo_link
    end

resulting in:

       Other specs relying on this:
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
       - RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
       - https://github.com/smiller/relies_on_example_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L4

Changes to the repo from this section are here.

At the end of the day, to go back to the original example of the developer from one team wanting to change a key from :breeding_stock to :bred_from, this doesn’t stop them from doing that. But it does make much more clear the other parts of the codebase (even from different repos) that rely on the assumption staying the same, and should prompt conversations before the change is made.

Coda: After Sleeping on It…

I greatly prefer the convenience of the line number / github link references in the other repos code to the ObjectSpace references in the same repo code. Everything described here works, but I wonder if it would be even better to start from greppable comment annotations instead of methods, like so:

# requirement anchor
# @REQUIREMENT: my_hounds includes key :breeding_stock

# relies_on anchor in same repo
# @RELIES_ON: my_hounds includes key :breeding_stock

# relies_on anchor in another repo
# @RELIES_ON: <repo:hounds_service>:my_hounds includes key :breeding_stock

See now List Tests Relying on Another Test: Another Approach, where I’ve developed this version as well.

Appendix: The Thread from November 2019

This idea started in a twitter thread I wrote in November 2019, and recently re-read, and decided it was still a good idea:

[Germ of a testing idea] Suppose you’ve got a test. It depends on something elsewhere in the system behaving in a certain way. Within the test, you stub out that behaviour. Which is fine, provided your assumption holds.

If the other behaviour changes without warning, your assumption and your test become worse than useless. To prevent that, you make sure there’s a test elsewhere in the system testing the behaviour you’re relying on, and you add it if it isn’t already there.

But, if that behaviour changes – not just the interface, which we can auto-check, but the behaviour – it would be useful if there were an automatic way of notifying “Now you need to check these 12 places to see if they still work because they assumed the previous behaviour”

And so, maybe in addition to the tests-in-two-places, we need a marker on the dependent test, a depends-on annotation, some sort of mechanism that if you change behaviour X, it can report or “potentially or secondarily fail” all the places Y that depend on X.

This isn’t going back to the brittle tests problem of “you make a change and 27 tests start breaking”, or at least it’s different in that it tells you the one place to look at first.

It probably also indicates, if 26 other places depend on the behaviour, it should be treated as a public interface and not changed or only changed very carefully. So probably, in addition to running tests, we’d want to be able to build reports from it.

This is indirectly down to @alex_schl, as I was watching her keynote on exploratory testing from https://twitter.com/alex_schl/status/1196804674373980166 while thinking about how to improve some tests I’d written yesterday. All flaws are of course my own, and I will try to write this up with more detail soon.