Problem Statement
Suppose you have three services, let’s call them hounds_service
, breeding_service
, and traits_service
. Suppose further that breeding_service
and traits_service
both call my_hounds
on hounds_service
, which returns something including (from Theseus’ speech in A Midsummer Night’s Dream):
{
breeding_stock: "Spartan",
traits: {
flewed_as: "Spartan",
sanded_as: "Spartan"
}
…
}
For within-the-service tests, breeding_service
and traits_service
will stub out these responses. But they’ll also want tests in hounds_service
that the values they care about appear in the actual response. If different teams own the different services, then, ideally, if the hounds_service
team decides to rename breeding_stock
to bred_from
for their own reasons, they should see the broken test and know they need to go talk with the other team whose test on their code they just broke to come to some agreement about how or whether the field name needs to change consistently.
The simplest thing would be to add a comment:
# if this breaks, please talk to the breeding_service team
describe "my_hounds response includes 'breeding_stock'" do
…
But it might be more efficient if there were some way of seeing at a glance which other tests (possibly on other repos) depend on this behaviour.
On the programming by wishful thinking approach, let’s introduce two new methods that we can put in spec blocks, requirement
and relies_on
. So:
# in hounds_service specs:
describe "my_hounds response includes 'breeding_stock'" do
def requirement
"my_hounds response includes 'breeding_stock'"
end
# in breeding_service specs:
describe "code processing the my_hounds response" do
def relies_on
"hounds_service:my_hounds response includes 'breeding_stock'"
end
Since the requirement
is the same as the describe
block’s description in this case, we might consider just using the description text, but starting with something that is more explicitly a search target, and less likely to be edited than a description, seems a safer first step.
Now we’ve got a concrete problem. Given a requirement
, we want to locate and list all the relies_on
blocks mentioning that requirement, either in the same repo or in other repos.
What we’ll want next is a relies_on
lookup, where the keys are the relies_on
text, and the values are the other spec example groups that have that text, and which we could interrogate as:
relied_on = relies_on_lookup.fetch(requirement, [])
if relied_on.any?
# other specs have relies_on text matching our spec's requirement text
# alert the user accordingly if our spec fails
end
(Edited to add: For a different approach, using comment annotations instead of method calls, see now also List Tests Relying on Another Test: Another Approach. But that very much builds on the work done here.)
In the Same Repo…
Let’s work with an example. Given a lib/service.rb
:
class Service
def my_hounds
{
breeding_stock: "Spartan",
traits: {
flewed_as: "Spartan",
sanded_as: "Spartan"
}
}
end
end
And a spec/with_requirement_spec.rb
:
require "service"
RSpec.describe "with requirement" do
subject { Service.new.my_hounds }
context "my_hounds" do
describe ":breeding_stock key" do
def requirement
"my_hounds includes :breeding_stock"
end
it "is included" do
expect(subject).to include :breeding_stock
end
end
describe ":traits key" do
def requirement
"my_hounds includes :traits"
end
it "is included" do
expect(subject).to include :traits
end
end
context "using my_hounds" do
describe "for :breeding_stock" do
def relies_on
"my_hounds includes :breeding_stock"
end
it "does something with :breeding_stock" do
expect(subject[:breeding_stock]).to eq("Spartan")
end
end
describe "for :traits" do
def relies_on
"my_hounds includes :traits"
end
it "does something with :traits" do
expect(subject[:traits][:flewed_as]).to eq("Spartan")
end
end
end
describe "using my_hounds :breeding_stock again" do
def relies_on
"my_hounds includes :breeding_stock"
end
it "checks something else" do
expect(subject[:breeding_stock]).not_to eq("Athenian")
end
end
end
end
That passes:
$ rspec
.....
Finished in 0.02194 seconds (files took 0.31486 seconds to load)
5 examples, 0 failures
And can be made to fail by commenting out line 5 of service
:
$ rspec
F.F..
Failures:
1) with requirement my_hounds :breeding_stock key is included
Failure/Error: expect(subject).to include :breeding_stock
expected {:traits => {:flewed_as => "Spartan", :sanded_as => "Spartan"}} to include :breeding_stock
Diff:
@@ -1 +1 @@
-[:breeding_stock]
+:traits => {:flewed_as=>"Spartan", :sanded_as=>"Spartan"},
# ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'
etc…
And we’d like the first failure message to include “Check out these other specs that rely on the assumption made in this one: …”.
Step 1: Build the relies_on lookup
Any spec file (or example group in a spec file) is going to be turned into a class extending RSpec::Core::ExampleGroup
. So we can, at the beginning of the spec run, read ObjectSpace
and find all the classes extending that:
ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
# do something with it
end
Note that we need to do this at the beginning of the test run because ObjectSpace
contains living objects and partway through the test run some of them might already be garbage collected and no longer there. (Yes, I have experimentally verified this.) We can put it into before(:suite)
, run once before everything else. We can’t save it to an instance variable from there, but we can add a setting to RSpec.configuration
:
config.add_setting :relies_on
config.before(:suite) do
config.relies_on = reckon_relies_on
end
which we can access later as RSpec.configuration.relies_on
.
In reckon_relies_on
, we need to grab all the RSpec classes, instantiate them, see if they contain / respond to relies_on
, and add them to the lookup if they do (note that multiple example groups might rely on the same requirement, like specs 3 and 5 above, so the value is an array)
def reckon_relies_on
relies_on = Hash.new { |hash, key| hash[key] = [] }
ObjectSpace.each_object(Class).select { |klass| klass < RSpec::Core::ExampleGroup }.each do |klass|
instance = klass.new
next unless instance.respond_to?(:relies_on)
relies_on_key = instance.method(:relies_on).call
relies_on[relies_on_key] << klass
end
relies_on
end
If we make those changes in spec/spec_helper.rb
we can put a binding.pry
in the first spec, and in the console see what RSpec.configuration.relies_on
is set to:
{
"my_hounds includes :breeding_stock" =>
[RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain,
RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock],
"my_hounds includes :traits" =>
[RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForTraits]
}
At which point we have both the requirement
(because we’re in the spec) and the relies_on
lookup to check it against. If the spec passes, there’s nothing to do. But if it fails, and other specs rely on the requirement, we want to display that.
Step 2: Display the relied_on related specs in the failure output
We can start by building the addition to the error message that we wish we had. Add another method to spec/spec_helper.rb
:
def relies_on_message(requirement)
relies_on = RSpec.configuration.relies_on.fetch(requirement, [])
if relies_on.any?
"\nOther specs relying on this: \n- #{relies_on.join("\n- ")}"
else
""
end
end
And now, from the binding.pry
in the first spec, if we call relies_on_message(requirement)
, we get:
Other specs relying on this:
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
To incorporate that into the actual error message, let’s build a custom matcher, which lets us customize the failure message. Add this as spec/support/matchers.rb
(also add require 'support/matchers'
to the top of spec/spec_helper.rb
so that it can be used):
RSpec::Matchers.define :include_key do |key, requirement|
match { |hash| hash.include?(key) }
failure_message do
"expected #{actual.keys} to include key #{key}, but it didn't\n- #{actual}\n#{relies_on_message(requirement)}"
end
failure_message_when_negated do
"expected #{actual.keys} not to include key #{key}, but it did\n- #{actual}\n#{relies_on_message(requirement)}"
end
end
And change the checks in the first two specs (lines 15 and 25) from (for line 15):
expect(subject).to include :breeding_stock
to
expect(subject).to include_key(:breeding_stock, requirement)
And then, if we comment out line 5 of lib/service.rb
to break the first spec, we get the error message including the desired “Other specs relying on this assumption” block:
rspec
F.F..
Failures:
1) with requirement my_hounds :breeding_stock key is included
Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)
expected [:traits] to include key breeding_stock, but it didn't
- {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}
Other specs relying on this:
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
# ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'
etc…
It is, granted, less than intuitive that we need to pass the requirement
into the matcher – include_key(:breeding_stock, requirement)
– what does requirement
have to do with whether :breeding_stock
is in my_hounds
is a valid question, and the answer is nothing, save that we need requirement
in there to check the relies_on lookup.
It is also the case that we’ll need to build other custom matchers for other kinds of checks if we want to include the relies_on_message(requirement)
for those too.
But this still feels like a step in a good direction. Example repo with all changes described so far here.
A Rejected Alternate Approach
This doesn’t require custom matchers but does require a rescue for each spec in a block with a requirement
, which felt worse, but, for completeness:
Add this to spec/spec_helper.rb
:
def append_to_failure_message(e, requirement)
e.message << relies_on_message(requirement)
raise e
end
And in the spec, change line 15 from
expect(subject).to include :breeding_stock
to
expect(subject).to include(:breeding_stock)
rescue RSpec::Expectations::ExpectationNotMetError => e
append_to_failure_message(e, requirement)
This does work, but using the rescue like this – even granted that it’s only going to happen when there’s a failing spec, so arguably it is a properly exceptional case instead of just control flow – didn’t sit well with me, hence the custom matcher approach I went with instead.
In Other Repos…
The simplest way to drive out the building of the other repos functionality is to create another repo with a spec with a relies_on
method referring back to a requirement
in this one:
def relies_on
"relies_on_example:my_hounds includes :breeding_stock"
end
Then, in our repo, we need a list of the related_repos we need to search, a reference to what the related repos are calling our repo, and a way of cloning the related repos into our repo so we can examine them. We can add this to spec/spec_helper.rb
:
def clone_related_repos
FileUtils.rm_r("related_repos") if Dir.exist?("related_repos")
FileUtils.mkdir("related_repos")
FileUtils.cd("related_repos")
related_repos.each do |repo|
system("git clone #{repo}")
end
end
def this_repo
"relies_on_example"
end
def related_repos
["https://github.com/smiller/relies_on_example_related_repo"]
end
We’ll also want to add related_repos/
to .gitignore
so we aren’t committing them.
Now if we call clone_related_repos
in reckon_relies_on
, we’ll have added a related_repos/
directory with the source code for all the related repos. We aren’t going to set them all running and examine their ObjectSpace
(in the real not-simplified-for-a-blog-post example at the back of this there are a dozen related repos), so we need something simpler and quicker.
At this point, I put a binding.pry
in reckon_relies_on
and started experimenting. Using open3
to capture the results of grepping for "relies_on_example:
stdout, _, _ = Open3.capture3("grep", "-nr", "\"#{this_repo}:", ".")
reveals
"./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4: \"relies_on_example:my_hounds includes :breeding_stock\"\n"
(In a more complex project, this check might not be distinct enough to avoid false positives: in that case, we would want to make the related repo relies_on
string more unique, for instance, <from:relies_on_example>:my_hounds include :breeding_stock
), and search for that instead.)
From that, after some trial and error, we can further process the results and add them to the relies_on
hash:
matches = stdout.split("\n")
matches.each do |line|
file, key = line.split(/\:\s+\"/)
key.sub!("#{this_repo}:", "")
.sub!("\"", "")
if file.include?("/spec/") && file.include?("_spec.rb:")
relies_on[key] << file
end
end
And then, if we comment out line 5 of lib/service.rb
and make the first spec fail again, we get a failure message including the line number of the relying-on spec in another repo:
1) with requirement my_hounds :breeding_stock key is included
Failure/Error: expect(subject).to include_key(:breeding_stock, requirement)
expected [:traits] to include key breeding_stock, but it didn't
- {:traits=>{:flewed_as=>"Spartan", :sanded_as=>"Spartan"}}
Other specs relying on this:
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
- ./relies_on_example_related_repo/spec/relies_on_related_repo_spec.rb:4
# ./spec/with_requirement_spec.rb:15:in `block (4 levels) in <top (required)>'
Further refinements are possible: if a github link to the line number in the other repo would be more likely to be followed up than a line number, we could change the code to provide that instead:
if file.include?("/spec/") && file.include?("_spec.rb:")
matches = /.\/(\w+)\/([\w\/.]+):(\d+)/.match(file)
repo_url = related_repos.select { |x| x[x.rindex('/') + 1..] == matches[1] }.first
repo_link = "#{repo_url}/blob/main/#{matches[2]}#L#{matches[3]}"
relies_on[key] << repo_link
end
resulting in:
Other specs relying on this:
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHoundsBreedingStockForAgain
- RSpec::ExampleGroups::WithRequirement::MyHounds::UsingMyHounds::ForBreedingStock
- https://github.com/smiller/relies_on_example_related_repo/blob/main/spec/relies_on_related_repo_spec.rb#L4
Changes to the repo from this section are here.
At the end of the day, to go back to the original example of the developer from one team wanting to change a key from :breeding_stock
to :bred_from
, this doesn’t stop them from doing that. But it does make much more clear the other parts of the codebase (even from different repos) that rely on the assumption staying the same, and should prompt conversations before the change is made.
Coda: After Sleeping on It…
I greatly prefer the convenience of the line number / github link references in the other repos code to the ObjectSpace references in the same repo code. Everything described here works, but I wonder if it would be even better to start from greppable comment annotations instead of methods, like so:
# requirement anchor
# @REQUIREMENT: my_hounds includes key :breeding_stock
# relies_on anchor in same repo
# @RELIES_ON: my_hounds includes key :breeding_stock
# relies_on anchor in another repo
# @RELIES_ON: <repo:hounds_service>:my_hounds includes key :breeding_stock
See now List Tests Relying on Another Test: Another Approach, where I’ve developed this version as well.
Appendix: The Thread from November 2019
This idea started in a twitter thread I wrote in November 2019, and recently re-read, and decided it was still a good idea:
[Germ of a testing idea] Suppose you’ve got a test. It depends on something elsewhere in the system behaving in a certain way. Within the test, you stub out that behaviour. Which is fine, provided your assumption holds.
If the other behaviour changes without warning, your assumption and your test become worse than useless. To prevent that, you make sure there’s a test elsewhere in the system testing the behaviour you’re relying on, and you add it if it isn’t already there.
But, if that behaviour changes – not just the interface, which we can auto-check, but the behaviour – it would be useful if there were an automatic way of notifying “Now you need to check these 12 places to see if they still work because they assumed the previous behaviour”
And so, maybe in addition to the tests-in-two-places, we need a marker on the dependent test, a depends-on annotation, some sort of mechanism that if you change behaviour X, it can report or “potentially or secondarily fail” all the places Y that depend on X.
This isn’t going back to the brittle tests problem of “you make a change and 27 tests start breaking”, or at least it’s different in that it tells you the one place to look at first.
It probably also indicates, if 26 other places depend on the behaviour, it should be treated as a public interface and not changed or only changed very carefully. So probably, in addition to running tests, we’d want to be able to build reports from it.
This is indirectly down to @alex_schl, as I was watching her keynote on exploratory testing from https://twitter.com/alex_schl/status/1196804674373980166 while thinking about how to improve some tests I’d written yesterday. All flaws are of course my own, and I will try to write this up with more detail soon.