Adding GitHub Actions to Run RSpec and SimpleCov

Summary

Given a simple Ruby project (say this one), how easy is it to set up a GitHub action on the repo so that it runs specs on push? And then say SimpleCov, too?

TL;DR: Skipping the trial and error and the credits (all below), just add this as .github/workflows/run-tests.yml to your repo and push it and you’re away:

name: Run tests
on: [push, pull_request]
jobs:
  run-tests:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          bundler-cache: true
      - name: Run tests
        run: bundle exec rspec -f j -o tmp/rspec_results.json -f p
      - name: RSpec Report
        uses: SonicGarden/rspec-report-action@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          json-path: tmp/rspec_results.json
        if: always()
      - name: Report simplecov
        uses: aki77/simplecov-report-action@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
        if: always()
      - name: Upload simplecov results
        uses: actions/upload-artifact@master
        with:
          name: coverage-report
          path: coverage
        if: always()

Adding RSpec

I did a search and found Dennis O’Keeffe’s useful article and since I already had a repo I just plugged in his .github/workflows/rspec.yml file:

name: Run RSpec tests
on: [push]
jobs:
  run-rspec-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          # Not needed with a .ruby-version file
          ruby-version: 2.7
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true
      - name: Run tests
        run: |
          bundle exec rspec

and it just ran, after three slight tweaks:

  • My repo already had a .ruby-version file so I could skip lines 11-12
  • the checkout@v2 action now gives a deprecation warning so I upped it to checkout@v4
  • because my local machine was a Mac I also needed to run bundle lock --add-platform x86_64-linux and push that for it the action to run on ubuntu-latest. Unsurprisingly.

The specs failed, which was unexpected, since they were passing locally, and it would be more informative if it named the specific failure on the summary page instead of needing me to click into run-tests to see it:

So I went to the GitHub Actions Marketplace and searched for “rspec”. There’s a RSpec Report action that looks like it does what we want if we change the bundle exec rspec line to output the results as a json file, so I changed the end of the workflow file to

      - name: Run tests
        run: bundle exec rspec -f j -o tmp/rspec_results.json -f p
      - name: RSpec Report
        uses: SonicGarden/rspec-report-action@v2
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          json-path: tmp/rspec_results.json
        if: always()

pushed, and tried again, and it failed again, but this time with details:

The failure is not github-action-specific (Digressions on the repo #1: RSpec): the TL;DR is that I had manually created a spec/tmp_files directory for files the tests wrote and because of the way I was doing setup and teardown it would have continued to work on my machine and nobody else’s without my noticing until I erased my local setup or tried it on a new machine. This was a very useful early warning.

Line 22’s if: always() is worth a mention. If running the tests (lines 15-6) exits with failing specs, subsequent blocks won’t run by default. If we need them to run anyway, we need to add if: always(), which is why it’s included here in all subsequent blocks.

Adding SimpleCov

Since the project wasn’t already using SimpleCov, step one was setting that up locally (Digressions on the repo #2: SimpleCov).

That sorted, I went back to the GitHub Actions Marketplace and searched for “simplecov” and started trying alternatives. The currently-highest-starred action, Simplecov Report, drops in neatly after running the specs and the RSpec Report:

      - name: Simplecov Report
        uses: aki77/simplecov-report-action@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

– it defaults to requiring 90% coverage, but you could pass in a different value by passing in a failedThreshold:

      - name: Simplecov Report
        uses: aki77/simplecov-report-action@v1
        with:
          failedThreshold: 80
          token: ${{ secrets.GITHUB_TOKEN }}

and of course if we want it to run whether specs fail or not, we need to add if: always().

So, I added

  - name: Simplecov Report
    uses: aki77/simplecov-report-action@v1
    with:
      token: ${{ secrets.GITHUB_TOKEN }}
    if: always()

and pushed. That passed:

That said, as with the spec results, it would be useful to see more detail. I remembered a previous project where we’d uploaded the SimpleCov coverage report to GitHub so it showed up as among the artifacts, and the steps in Jeremy Kreutzbender’s 2019 article still work for that: we can add a fourth block to our file:

- name: Upload coverage results
  uses: actions/upload-artifact@master
  with:
    name: coverage-report
    path: coverage
  if: always()

and push, and that gives us access to the coverage report:

which we can download and explore the details down to line numbers, even if our local machine is on a different branch.

And, for pushes, we’re done. But there’s two more things we need to add for it to work with pull requests as well.

For Pull Requests

The first thing to do is to modify the action file so that the we trigger the workflow on pull requests as well as pushes:

on: [push, pull_request]

At which point, it feels like everything should work, but if we create a PR, the SimpleCov Report fails, with Error: Resource not accessible by integration:

It ran the code coverage, as we can see from the end of the Run tests block, it uploads the coverage results, but the SimpleCov Report block fails.

This is because, in the PR case, the SimpleCov Report tries to write a comment to the PR with its coverage report:

and to enable it to do that, we need to give it permission, so we need to insert into the action file:

permissions:
  contents: read
  pull-requests: write

And that fixes the SimpleCov Report error and gets us to a passing run.

I was initially unwilling to do this because pull-requests: write felt like a dangerously wide-scoped permission to leave open, but, on closer examination, the actual permissions that it enables are much more narrowly scoped, to issues and comments and labels and suchlike.

(Note that we only need to add these permissions because SimpleCov Report is adding a comment to the PR: if we’d picked a different SimpleCov reporting action that didn’t write a comment, we would only have needed on: [push, pull_request] to get the action to work on pushes and pull requests.)

This was an interesting gotcha to track down, but now we’re done, for both pushes and pull requests.

With thanks to my friend Ian, who unexpectedly submitted a pull request for the repo which exposed the additional wrinkles that had to be investigated here.

Appendix: Digressions on the Repo

#1: RSpec

The first time I ran the RSpec action it failed, which usefully revealed that the setup in one of my tests was relying on a manual step.

The repo takes an input file and writes an output file, so in spec/ I’ve got spec/example_files for the files it starts from and spec/tmp_files for the new files it writes. I had created that directory locally and was running

  before do
    FileUtils.rm_r("spec/tmp_files")
    FileUtils.mkdir("spec/tmp_files")
  end

Which only worked, of course, because before the first run I’d manually added the directory, which I didn’t remember until the GitHub action tried and failed because it had no idea about that manual step.

The simplest fix would be to replace it with

  before do
    FileUtils.mkdir("spec/tmp_files")
  end

  # ... tests

  after do
    FileUtils.rm_r("spec/tmp_files")
  end

but because I was using the specs to drive the implementation and I was eyeballing the produced files as I went, I didn’t want to delete them in teardown, so I did this instead:

  before do
    FileUtils.rm_r("spec/tmp_files") if Dir.exist?("spec/tmp_files")
    FileUtils.mkdir("spec/tmp_files")
  end

after which both the local and the GitHub action versions of the tests passed. This discovery was an unexpected but very welcome benefit of setting up the GitHub action to run specs remotely.

Back

#2: SimpleCov

To add SimpleCov locally, we add gem "simplecov" to the Gemfile, require and start it from the spec/spec_helper.rb file (filtering out spec files from the report):

require 'simplecov'
SimpleCov.start do
  add_filter "/spec/"
end

and add coverage/ to the .gitignore file so we aren’t committing or pushing the files from local coverage runs.

Then we can run RSpec locally, open the coverage/index.html file, and discover that though we know that both lib/ files are being exercised in the specs, only one of them shows up in the coverage report:

And this makes sense, unfortunately, because lib/formatter.rb contains the class which does the formatting, and lib/reformat.rb is a script for the command-line interface which is largely optparse code and param checking. But it makes for a misleading coverage report.

We can start fixing this by moving everything but the command-line interface and optparse code out of the lib/reformat.rb into, say, lib/processor.rb, and have the CLI call that. It still won’t show up in the coverage report because it isn’t being tested directly, but we can add tests against Processor directly so that they do.

Having done that, we get a much more satisfactory coverage report:

This still leaves a very small bit of code in the CLI that isn’t covered by the coverage report:

option_parser.parse!
variant = ARGV[0]

if (variant.nil? && options.empty?) || variant == "help"
  puts option_parser
end

if variant == "alternating"
  Processor.new(options: options).process
end

but we have tests that will break if that doesn’t work so we decide we can live with that. (If it got more complicated, with multiple variants calling multiple processors, we could pull lines 4-10 into their own class and test it directly too. But it hasn’t yet.)

Back

Adding described_url to RSpec for Request Specs

Background

RSpec has a described_class method, which, given a spec starting

RSpec.describe Accounting::Ledger::Entry do

lets us use described_class for Accounting::Ledger::Entry within the spec, which is less noisy (particularly with long class names) and very handy if we later rename the class.

RSpec also has a subject method which exposes not the class but an instance of the class. It can be over-ridden implicitly, by putting a different class name in an inner describe block’s description:

RSpec.describe Accounting::Ledger::Entry do
  # subject starts as instance of Accounting::Ledger::Entry
  …
  describe Accounting::Ledger::SpecificEntryType do
    # subject is now instance of Accounting::Ledger::SpecificEntryType

or explicitly, by defining it directly:

RSpec.describe Accounting::Ledger::Entry do
  # subject starts as instance of Accounting::Ledger::Entry
  …
  subject { "just what I choose it to mean – neither more nor less" }
  # subject is now that string, just like Humpty Dumpty said

Problem

What if we’re building a request spec for an API endpoint, and our spec starts

RSpec.describe "/accounting/ledger/entries" do
  it "GET" do
    get "/accounting/ledger/entries"

    expect(response).to have_http_status(:ok)

It would be nicer if we had a described_url method, so we could say get described_url instead of repeating the url string in each example.

We could set subject once to the url:

RSpec.describe "/accounting/ledger/entries" do
  subject { "/accounting/ledger/entries" }
  it "GET" do
    get subject

But the single repetition still feels like it should be unnecessary, particularly with the string we need only one line away.

Towards a Solution

We see from the documentation that we can get the inner description string from the example if we specify it as a block parameter:

RSpec.describe "/accounting/ledger/entries" do
  it "GET" do |example|
    expect(example.description).to eql("GET") # => passes
  end
end

We won’t want to say do |example| all the time, though, so let’s start building code in a single before block. Further, we’ll only want this in request specs, so let’s tag request specs as such

RSpec.describe "/accounting/ledger/entries", :requests do

(:requests here is equivalent to requests: true), and put the before do |example| block, scoped to specs tagged :requests, in spec/spec_helper.rb:

RSpec.configure do |config|
  …
  config.before(:each, :requests) do |example|
    binding.pry
  end

From here we can run the spec and at the debugger and see that example.description is indeed “GET”.

> example
=> #<RSpec::Core::Example "GET">
> example.description
=> "GET"

Looking at lib/rspec/core/example.rb shows us that def description looks in something called metadata[:description] and passing example.metadata to the debugger gives us a screen-full of hash data, including

{ … 
:description => "GET"
:full_description => "/accounting/ledger/entries GET",
:example_group => { …
  :description => "/accounting/ledger/entries"

So if we’re one layer deep, example.metadata[:example_group][:description] gives us the outer description.

However, the actual example might be nested inside a describe or context block. Given

RSpec.describe "/accounting/ledger/entries" do
  describe "GET" do
    it "first case" do

the relevant metadata extract is instead:

{ … 
:description => "first case"
:full_description => "/accounting/ledger/entries GET first case",
:example_group => { …
  :description => "GET",
  :full_description => "/accounting/ledger/entries GET",
  …
  :parent_example_group => { …
    :description => "/accounting/ledger/entries",
    :full_description => "/accounting/ledger/entries"

And yes, testing with another nested group demonstrates that a :parent_example_group can include an inner :parent_example_group.

A Solution

But that’s a simple recursion problem: keep going until you don’t have a next :parent_example_group and then grab the outermost :description. We could replace the contents of the before block with:

def reckon_described_url(metadata)
  if metadata[:parent_example_group].nil?
    metadata[:description]
  else
    reckon_described_url(metadata[:parent_example_group])
  end
end

@described_url = reckon_described_url(example.metadata[:example_group])

Why the Snark Is a Boojum

(Yes, arguably, because the first space in the :full_description string happens to be between the outer description and the second-outermost description, we could get the same answer from

@described_url = example.metadata[:full_description].split(" ").first

but only by coincidence, and it would break for unrelated reasons if they changed how they build :full_description. So let’s not do that.)

Back to the Solution

We can now use the code from our before blog in a request spec. Going back to our original example, we can now implement it

RSpec.describe "/accounting/ledger/entries", :requests do
  it "GET" do
    get @described_url

If we’d prefer not to have an unexplained instance variable, or are looking forward to the possibility of passing in parameters, we can wrap it in a method call in the before block:

def described_url
  @described_url
end

And then it can (finally) be used by the end user just like described_class:

RSpec.describe "/accounting/ledger/entries", :requests do
  it "GET" do
    get described_url

A Solution that Takes Parameters

A more realistic version of the request spec url might be /accounting/ledgers/:ledger_id/entries. Fortunately, since we’ve just turned described_url into a method, we can make it take parameters, so a slightly changed test can pass in the parameters it needs to substitute:

RSpec.describe "/accounting/ledgers/:ledger_id/entries", :requests do
  let(:ledger) { … build test ledger … }
  it "GET" do
    get @described_url(params: {ledger_id: ledger.id})

and the method can create a copy of the base @described_url (as different tests may well require different parameters) and return the copy with the parameters, if any, filled in:

def described_url(params: {})
  url = @described_url.dup
  params.each do |key, value|
    url.sub!(":#{key}", value.to_s)
  end
  url
end

Appendix: The Complete Before Block

RSpec.configure do |config|
  …
  config.before(:each, :requests) do |example|
    def described_url(params: {})
      url = @described_url.dup
      params.each do |key, value|
        url.sub!(":#{key}", value.to_s)
      end
      url
    end

    def reckon_described_url(metadata)
      if metadata[:parent_example_group].nil?
	    metadata[:description]
      else
	    reckon_described_url(metadata[:parent_example_group])
      end
    end

    @described_url = reckon_described_url(example.metadata[:example_group])
  end
  …
end

Testing with Page Objects: Implementation

In the previous post, we added the page object framework SitePrism to a project, did some organization and customization, and added helper methods so we could call visit_page and on_page to interact with the app. Next, let’s start using it.

Navigation

First Step

Let’s say that for the first test you want to go to the login page, log in, and then assert that you’re on the dashboard page. That’s easy enough to do like so:

visit_page("Login").login(username, password)
on_page("Dashboard")

module PageObjects
  class Login < SitePrism::Page
    set_url "/login"

    element :username, "input#login"
    element :password, "input#password"
    element :login, "input[name='commit']"

    def login(username, password)
      self.username.set username
      self.password.set password
      login.click
    end
  end
end

module PageObjects
  class Dashboard < SitePrism::Page
    set_url "/dashboard"
  end
end

The first line will go to the “/login” page and then try to submit the form, and will fail if it can’t find or interact with the “username”, “password”, or “login” elements. For instance, if the css for the username field changes, the test will fail with:

Unable to find css "input#login" (Capybara::ElementNotFound)
./test_commons/page_objects/login.rb:10:in `login'
./features/step_definitions/smoke.rb:7:in `login'
./features/step_definitions/smoke.rb:3:in `/^I log in$/'
features/smoke.feature:5:in `When I log in’

The second line will fail if the path it arrives at after logging in is something other than “/dashboard”.

Because the default message for the different path was uninformative (expected true, but was false), we added a custom matcher to provide more information. The default message for not finding an element on the page, including the css it’s expecting and the backtrace to the page object it came from and the line in the method it was trying to run, seems sufficiently informative to leave as-is.

If the login field were disabled instead of not found, the error message would be slightly less informative:

invalid element state
  (Session info: chrome=…)
  (Driver info: chromedriver=2.14. … (Selenium::WebDriver::Error::InvalidElementStateError)
./test_commons/page_objects/login.rb:10:in `login'
./features/step_definitions/smoke.rb:7:in `login'
./features/step_definitions/smoke.rb:3:in `/^I log in$/'
features/smoke.feature:5:in `When I log in’

but since the backtrace still gives you the line of the page object where it failed, and that gets you the element in an invalid state, it feels lower priority.

Second Step

If the next thing you want to test is a subcomponent flow, you can navigate to its first page either by calling visit_page('ComponentOne::StepOne') (the equivalent of typing the path in the browser address bar), or by having the test interact with the same navigation UI that a user would. The second choice feels more realistic and more likely to expose bugs, like the right navigation links not being displayed.

(Ironically, having created the visit_page method, we might only ever use it for a test’s first contact with the app, using in-app navigation and on_page verification for all subsequent interactions.)

To get that to work, we need to model the navigation UI.

SitePrism Sections and Navigation UI

Supose the app has a navigation bar on every logged-in page. We could add it as a section to the Dashboard, say, but if it’s on every page it probably makes more sense to add it as a section to a BasePage and have Dashboard (and all other logged-in pages) inherit from it:

module PageObjects
  class BasePage < SitePrism::Page
    section :navigation, Navigation, ".navigation"
  end
end

module PageObjects
  class Dashboard < BasePage
    …
  end
end

module PageObjects
  class Navigation < SitePrism::Section
    element :component_one, ".component_one"
    element :componont_two, ".component_two"

    def goto_component_one
      component_one.click
    end

    def goto_component_two
      component_two.click
    end
  end
end

Now, if we want to go from the dashboard to the start of component one, we can modify the on_page("Dashboard") line above to

on_page("Dashboard").navigation.goto_component_one

Can we delegate to the section from the enclosing page? Yes. If we add this to the base page:

module PageObjects
  class BasePage < SitePrism::Page
    …
    delegate :goto_component_one, :goto_component_two, to: :navigation
  end
end

we can then call, still more briefly:

on_page("Dashboard").goto_component_one

And now the test from the top can continue like so:

visit_page("Login").login(username, password)
on_page("Dashboard").goto_component_one
on_page("ComponentOne::StepOne").do_something

and line two will fail if the link to “component one” isn’t visible on the screen, and line three will fail if the user doesn’t end up on the first page of component one (perhaps they followed the link but authorizations aren’t set up correctly, so they end up on an “unauthorized” page instead).

What Else Is on the Page?

If you call a method on a page object it will verify that any item that it interacts with is on the page, and fail if it can’t find it or can’t interact with it. But what if you’re adding tests to an existing app, and you aren’t yet calling methods that interact with everything on the page, but you do want to make sure certain things are on the page?

Suppose you have a navigation section with five elements, unimaginatively called :one, :two, :three, :four, :five. And you have three types / authorization levels of users, like so:

User Can see elements
user1 :one
user3 :one, :two, :three
user5 :one, :two, :three, :four, :five

And you want to log each of them in and verify that they only see what they should see. We can do this with SitePrism’s has_<element>? and has_no_<element>? methods.

A naive approach might be:

all_navigation_elements = %w{one two three four five}

visit_page('Login').login(user1.username, user1.password)
on_page('Dashboard') do |page|
  user1_should_see = %w{one}
  user1_should_see.each do |element|
    expect(page.navigation).to send("have_#{element}")
  end
  user1_should_not_see = all_navigation_elements - user1_should_see
  user1_should_not_see.each do |element|
    expect(page.navigation).to send("have_no_#{element}")
  end
end

Even before doing the other two, it’s clear that we’re going to want to extract this into a more generalized form.

To build out the design by “programming by wishful thinking”, if we already had the implementation we might call it like this:

all_navigation_elements = %w{one two three four five}

visit_page('Login').login(user1.username, user1.password)
on_page('Dashboard') do |page|
  should_see = %w{one}
  has_these_elements(page.navigation, should_see, all_elements)
end

visit_page('Login').login(user3.username, user3.password)
on_page('Dashboard') do |page|
  should_see = %w{one two three}
  has_these_elements(page.navigation, should_see, all_elements)
end

# do. for user5

And then we can reopen test_commons/page_objects/helpers.rb to add an implementation:

def has_these_elements(page, has_these, all)
  has_these.each do |element|
    expect(page).to send("have_#{element}")
  end
  does_not_have = all - has_these
  does_not_have.each do |element|
    expect(page).to send("have_no_#{element}")
  end
end

And this works. The raw error message (suppose we’re doing this test first so we’ve got the test for user1 but user1 still sees all five items) is unhelpful:

expected #has_no_two? to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:28:in `block in has_these_elements'
./test_commons/page_objects/helpers.rb:27:in `each'
./test_commons/page_objects/helpers.rb:27:in `has_these_elements'
./features/step_definitions/navigation_permissions.rb:24:in `block in dashboard'
…

And we could build a custom matcher for it, but in this case since the error is mostly likely to show up (if we’re working in Cucumber) after

When user1 logs in
Then user1 should only see its subset of the navigation

seeing expected #has_no_two? after that seems perfectly understandable.

Summing Up So Far

I’m quite liking this so far, and I can easily imagine after I’ve used them for a bit longer bundling the helpers into a gem to make it trivially easy to use them on other projects. And possibly submitting a PR to suggest adding something like page.has_these_elements(subset, superset) into SitePrism, because that feels like a cleaner interface than has_these_elements(page, subset, superset).

Testing with Page Objects: Setup

What Are Page Objects?

Page objects are an organizing tool for UI test suites. They provide a place to identify the route to and elements on a page and to add methods to encapsulate paths through a page. Martin Fowler has a more formal description.

Suppose you’re testing a web app which has a component with a four-page flow, with a known happy path a user would normally work through. Using page objects, you could test that flow as:

def component_one_flow(user, interests, contacts)
  goto_component_one_start
  on_page('ComponentOne::StepOne').register(user)
  on_page('ComponentOne::StepTwo').add_interests(interests)
  on_page('ComponentOne::StepThree').add_contacts(contacts)
  on_page('ComponentOne::StepFour').approve
end

If anything changes on one of those pages, you can make the change inside the page object, and the calling code can remain the same.

For small- to medium-sized web apps this might seem a nice-to-have, but for larger apps where you end up afraid to open features/step_definitions or spec/features because it’s impossible to find anything, this is a very useful pattern to introduce, or to start out with on the next project.

This ran quite long, so I’ve split it into two parts: in this first post I’ll describe setting up a page object framework, and in the second I’ll talk about implementation patterns.

Available Frameworks

In 2013 and 2014 I enjoyed working with Jeff Morgan’s page-object, which runs on watir-webdriver. (His book Cucumber and Cheese was my introduction to page objects.) Jeff Nyman’s Symbiont also runs on watir-webdriver and also looks quite interesting (as does his testing blog). There’s also Nat Ritmeyer’s SitePrism, running on Capybara.

Here’s a login page in all three:

Page-object

class LoginPage
  include PageObject

  page_url "/login"

  text_field(:username, id: 'username')
  text_field(:password, id: 'password')
  button(:login, id: 'login')

  def login_with(username, password)
    self.username = username
    self.password = password
    login
  end
end

SitePrism

class Login < SitePrism::Page
  set_url "/login"

  element :username, "input#login"
  element :password, "input#password"
  element :login, "input[name='commit']"

  def login_with(username, password)
    self.username.set username
    self.password.set password
    login.click
  end
end

Symbiont

class Login
  attach Symbiont

  url_is      "/login"
  title_is    "Login"

  text_field :username, id: 'user'
  text_field :calendar, id: 'password'
  button     :login,    id: 'submit'

  def login_with(username, password)
    self.username.set username
    self.password.set password
    login.click
  end
end

The similarities are clear: in each case you can identify the URL, the elements, and methods that interact with those elements. And since it is the internal page object names for elements that are used in the methods, if (when) the css changes, the only line that needs to change in the page object is the declaration of the element.

What about differences? Page-object (here) and Symbiont (here) provide page factories, so that in page-object you can write

visit(LoginPage) do |page|
  page.login(username, password)
end

or, even more compactly

visit(LoginPage).login(username, password)

where SitePrism is, somewhat more long-windedly:

page = Login.new
page.load
page.login(username, password)

SitePrism offers, in addition to element, the named concepts of elements (for a collection of rows, say) and section (for a subset of elements on a page, or elements which are common across multiple pages, a navigation bar, say, which you can identify separately in a SitePrism::Section object).

The further point that inclined me to try SitePrism (though with a mental note to add a page factory) was that when we were using page-object last year we sometimes needed to write explicit wrappers for waiting code because the implicit when_present wait did not always seem to be reliable, and it looked as though Symbiont wrapped it the same way (here). I was curious to see if SitePrism (or its underlying driver) had a more robust solution.

Setting up SitePrism

So How Does It Handle Waiting?

By default, SitePrism requires explicit waits, so either wait_for_<element> or wait_until_<element>_visible. This has been challenged (issue#41, in which Ritmeyer explains that he likes having more fine-grained control over timing of objects than Capybara’s implicit waits allows), and a compromise has been implemented (issue#43), by which you can configure SitePrism to use Capybara’s implicit waits, so:

SitePrism.configure do |config|
  config.use_implicit_waits = true
end

I’ve been using that, and it’s working so far.

Where to Put Them

On the earlier project we were using page objects from Cucumber, so we put them under features/page_objects. The new project is larger and already has non-test-framework-specific code under both features and spec, some of which is called from the other framework’s directory, so it makes sense to create a common top-level directory for them. I called it test_commons to make it clear at a glance through the top-level directories that it was test-related instead of app-related. Then page_objects goes under that.

The disadvantage of the location is that we will need to load the files in explicitly. So we add this at test_commons/all_page_objects.rb (approach and snippet both from Panayotis Matsinopoulos’s useful post):

page_objects_path = File.expand_path('../page_objects', __FILE__)
Dir.glob(File.join(page_objects_path, '**', '*.rb')).each do |f|
  require f
end

and in features/support/env.rb we add:

require_relative '../../test_commons/all_page_objects'

and we’re away. (If/when we use them from RSpec features as well, we can require all_page_objects in spec_helper.rb too.)

Further Organization

Probably you’ll have some top-level or common pages, like login or dashboard, and others which belong to specific flows and components, like the four pages of “component one” in the first example. A login page could go in the top-level page objects directory:

# test_commons/page_objects/login.rb
module PageObjects
  class Login < SitePrism::Page
    …
  end
end

whereas the four pages of component one could be further namespaced:

# test_commons/page_objects/component_one/step_one.rb
module PageObjects
  module ComponentOne
    class StepOne < SitePrism::Page
      …
    end
  end
end

This becomes essential when you’ve got more than a handful of pages.

Adding a Page Factory for SitePrism

I started by looking at page-object’s implementation (which itself references Alister Scott’s earlier post).

That seemed to do more than I needed right away, so I started with the simpler:

# test_commons/page_objects/helpers.rb
module PageObjects
  def visit_page(page_name, &block)
    Object.const_get("PageObjects::#{page_name}").new.tap do |page|
      page.load
      block.call page if block
    end
  end
end

… though we will need to add World(PageObjects) to our features/support/env.rb to use this from Cucumber (the require got us the page object classes, but not module methods). If we were running from RSpec, to borrow a snippet from Robbie Clutton and Matt Parker’s excellent LA Ruby Conference 2013 talk, we would need:

RSpec.configure do |c|
  c.include PageObjects, type: [:feature, :request]
end

So now we can call visit_page (not visit because that conflicts with a Capybara method), it’ll load the page (using the page object’s set_url value), and if we have passed in a block it’ll yield to the block, and either way it’ll return the page, so we can call methods on the returned page. In other words: visit_page('Login').login(username, password).

We can use the same pattern to build an on_page method, which instead of loading the page asserts that the app is on the page that the test claims it will be on:

def on_page(name, args = {}, &block)
  Object.const_get("PageObjects::#{page_name}").new.tap do |page|
    expect(page).to be_displayed
    block.call page if block
  end
end

Further iteration (including the fact that both page.load and page.displayed? take arguments for things like ids in URLs) results in something like this:

module PageObjects
  def visit_page(name, args = {}, &block)
    build_page(name).tap do |page|
      page.load(args)
      block.call page if block
    end
  end

  def on_page(name, args = {}, &block)
    build_page(name).tap do |page|
      expect(page).to be_displayed(args)
      block.call page if block
    end
  end

  def build_page(name)
    name = name.to_s.camelize if name.is_a? Symbol
    Object.const_get("PageObjects::#{name}").new
  end
end

We haven’t implemented page-object’s if_page yet (don’t assert that you’re on a page and fail if you aren’t, but check if you’re on a page and if you are carry on), but we can add it later if we need it.

What Would that Failed Assertion Look Like?

Suppose we tried on_page('Dashboard').do_something when we weren’t on the dashboard. What error message would we get?

expected displayed? to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:11:in `block in on_page'

We know, because SitePrism has a #current_path method, what page we’re actually on. A custom matcher to produce a more informative message seems a good idea. Between the rspec-expectations docs and Daniel Chang’s exploration, let’s try this:

# spec/support/matchers/be_displayed.rb
RSpec::Matchers.define :be_displayed do |args|
  match do |actual|
    actual.displayed?(args)
  end

  failure_message_for_should do |actual|
    expected = actual.class.to_s.sub(/PageObjects::/, '')
    expected += " (args: #{args})" if args.count > 0
    "expected to be on page '#{expected}', but was on #{actual.current_path}"
  end
end

which we’d get for free from RSpec, and from Cucumber we can get by adding to features/support/env.rb:

require_relative '../../spec/support/matchers/be_displayed'

The first time I ran this, because I wasn’t passing in arguments to SitePrism’s #displayed? method, I hit a weird error:

comparison of Float with nil failed (ArgumentError)
./spec/support/matchers/be_displayed.rb:3:in `block (2 levels) in <top (required)>'

… emending the helper methods to make sure that I included an empty hash even if no arguments were passed in (part of the final version of helpers.rb above) fixed that. Now, with the new matcher working, if we run on_page('Dashboard').do_something when we’re actually at the path /secondary-control-room, we get the much more useful error

expected to be on page 'Dashboard', but was on '/secondary-control-room' (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:11:in `block in on_page'

… note that if there had been expected arguments, we would have printed those out too (expected to be on page 'Dashboard' (args: {id: 42})).

I built the custom matcher in RSpec because that’s what I tend to use, but if Minitest is more your thing, you could add a custom matcher by reopening Minitest::Assertions. (Example here.)

Setup Complete

We’ll break now that the page object framework is set up, and discuss further implementation patterns in the next post.

Emacs Org-mode Styling, Non-smart Quotes, Zero-width-space, and TeX Input Method

In org-mode we can style inline elements with *bold* (bold), /italic/ (italic), _underlined_ (_underlined_, which won’t show up in html), =verbatim= (verbatim), and ~code~ (code).

But this breaks if the character just inside the styling code is a non-smart single or double quote. =C-c ;= is styled (C-c ;); =C-c '= is not (=C-c '=).

We can fix that by inserting a zero-width space between the apostrophe and the = . The first time, we can put the cursor between the apostrophe and the = and enter C-x 8 RET ZERO WIDTH SPACE RET, at which point =C-c '​= will display as C-c '.

The ZERO WIDTH SPACE part of insert-char (C-x 8) is autocompleting, so we won’t need to type the whole thing, but we’ll probably still want a shortcut if it we find ourselves doing it more than once or twice. Here’s the simplest thing we could do:

(defun my/insert-zero-width-space ()
(interactive)
(insert-char ?\u200B)) ;; code for ZERO WIDTH SPACE
(global-set-key (kbd "C-x 8 s") ‘my/insert-zero-width-space)

After which, if we find ourselves adding inline styles which don’t work because the styled section starts or ends with a quotation mark, we can go to just before or just after the quotation mark, type C-x 8 s, and watch the styles apply as expected.

Thanks, as usual, to Sacha Chua, who explained this in passing. Her .emacs has a more flexible solution which I will probably switch to the moment I need to insert more than one unicode character.

Note that while you can use insert-char to get accented characters (C-x 8 ' e produces é, for instance), and I used to do that, I have since set the input method to TeX (M-x set-input-method RET TeX RET), and now I type \​'e for é, \​th for þ, \​Mu\​epsilon\​nu for Μεν, and so on and so forth.

Automating Boilerplate in Org-mode Journalling

The Problem

My main org-mode files have a header structure like this:

* 2015
** March
*** Friday 6 March 2015
**** 10:00. Emacs coaching with Sacha :emacs:
<2015-03-06 10:00-11:00>
**** 23:37. Emacs progress note
[2015-03-06 23:37]
*** Saturday 7 March 2015
**** 12:55. Stratford King Lear in cinemas :wcss:
<2015-03-07 12:55-15:55>

The two things I do most often are start writing notes in the present, and set up scheduled events in the future. Scheduled events need active timestamps so that they show up in the agenda view:

Week-agenda (W10):
Monday 2 March 2015 W10
Tuesday 3 March 2015
Wednesday 4 March 2015
Thursday 5 March 2015
Friday 6 March 2015
10:00. Emacs Coaching with Sacha :emacs:
Saturday 7 March 2015
12:55. Stratford King Lear in cinemas :wcss:
Sunday 8 March 2015

[How to get the agenda view? Make sure the file you’re in is in org-agenda-files by pressing C-c [, then run org-agenda either by M-x org-agenda or the common (but not out-of-the-box) shortcut C-c a, then press a to get the agenda for the week.]

[Active timestamps are in < >; inactive timestamps are in [ ]. If we want to promote an inactive timestamp to be active and show up in the agenda view, we can run M-x org-toggle-timestamp-type from the inactive timestamp.]

[Start time is duplicated between header and timestamp deliberately: sometimes I review the file in org-mode outline with only the headers. If not for that use-case it would make sense to leave the start time out of the header.]

What I’d like to be able to do is open the file, and then with as little ceremony or setup as possible, start typing.

Adding a New Note

When I’ve already got a date header for the current date, this is trivially simple: I can put my cursor under that date and call M-x my note, having previously defined

(defun my/note ()
(interactive)
(insert "\n\n**** " (format-time-string "%H:%M") ". ")
(save-excursion
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n")))

… and it does pretty much what it says.

The values passed into format-time-string (%Y, %m, %d etc.) will be the values for the current date and time.

save-excursion means “do what’s inside here and then return the point (cursor) to where it was before the block was called”, so it fills in the timestamp and then returns the cursor to the end of the header line, ready for the title to be filled in.

Since after that I’d need to move the cursor back down to below the timestamp, I can be even lazier and prompt for the title and the tags (if any) and fill them in and move the cursor down to where the text should start:

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

If we were just setting one argument, say title, this would be simpler:

(defun my/note-2-eg (title)
(interactive "MTitle? ")

But since we want two, we need to tell interactive it’s getting a list, and then the arguments are set in order.

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))

Just as we can create an inactive timestamp by inserting a formatted date with [ ], we can add tags by inserting a string surrounded by : :. We don’t want to do this if the tag argument was left blank, of course, so we surround it with an unless:

(unless (string= tags "")
(insert " :" tags ":")
)

[Side-note: if you’re doing anything more complicated than this with strings, you probably want Magnar Sveen’s s.el (string manipulation library), which here would let you do (unless (s-blank? tags)… instead.]

At the end of which, because we took out the save-excursion, the cursor is below the timestamp and I’m ready to type.

All good. What if we don’t already have a date header for the current date? Can we auto-generate the date header?

Auto-generating Date Headers

Org-mode has a function called org-datetree-find-date-create. If you pass it in a list of numbers (the month, the day, and the year), and if your header structure is

* 2015
** 2015-03 March
*** 2015-03-06 Friday

then if you called that function passing in (3 7 2015), it would automatically add:

*** 2015-03-07 Saturday

For that matter, if you called it passing in (4 1 2015), it would automatically add

** 2015-04 April
*** 2015-04-01 Wednesday

If you call it passing in a date which is already there, it moves the cursor to that date. So, we could change the format of the org file headers, update our new note function to call the function, and be done.

(defun my/note-3 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

In the new bit:

(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)

format-time-string “%Y-%m-%d” will return “2015-03-07”, and org-date-to-gregorian will turn that into (3 7 2015), which is the format that org-datetree-find-date-create expects. Determining this involved looking at the source for org-datetree-find-date-create to see what arguments it expected (C-h f org-datetree-find-date-create takes you to a help buffer that links to the source file, in this case org-datetree.el; click on the link to go to the function definition) and a certain amount of trial and error. At one point, before org-date-to-gregorian, I had the also working but rather less clear:

(org-datetree-find-date-create (mapcar ‘string-to-number
(split-string (format-time-string "%m %d %Y"))))

org-end-of-subtree just takes the cursor to the bottom of the section for the date.

And that then works. What about adding new events in the future?

Adding a New Event

org-datetree-find-date-create makes it easier to fill in missing month and date headers to create a new future event:

(defun my/event (date end-time)
(interactive (list
(org-read-date)
(read-from-minibuffer "end time (e.g. 22:00)? ")))
(org-datetree-find-date-create (org-date-to-gregorian date))
(goto-char (line-end-position))
(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")
(save-excursion
(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n")))

There’s a problem I haven’t solved here, which is that org-read-date brings up the date prompt and lets you select a date, including a time or time range. If you select a time, it will be included in the date. If you select a time range, say 19:30-22:30, it ignores the time and the date object returned uses the current time. That’s not what we want.

So when the date prompt comes up:

Date+time [2015-03-07]: _ => <2015-03-07 Sat>

I can put in a new date and start time, say “2015-04-01 11:00”:

Date+time [2015-03-07]: 2015-04-01 11:00_ => <2015-04-01 Wed 11:00-14:00>

and then press enter and that gives me the date and the start time in a single string. We extract the start time and put it into the header like so:

(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")

Where split-string splits the date “2015-04-01 11:00” into a two-element list (“2015-04-01” “11:00”) {n}, and nth 1 returns the second element (list elements, here as elsewhere, are numbered starting from 0), or “11:00” {n}.

If the date had been “2015-04-01” without a start time, line 1 would have set start-time to nil, which would have blown up as an argument to insert (with a “Wrong type argument: char-or-string-p, nil”), so we check for a nil and set it to an empty string instead.

For the agenda view we’ll need the end-time as well, so we grab that as a second argument, and reassemble the date, say “2015-04-01 11:00”, and end-time, say “14:00”, into the active timestamp:

(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n"))

though here again, if we had just pressed enter when prompted for end time we would end up with an empty string and an invalid active timestamp, like “<2015-04-01->”, so check whether end-time has been set and build the timestamp string accordingly.

And since this is a future event and we’re probably only filling in the title, we’ll use save-excursion again to fill in the active timestamp and then go back and leave the cursor on the header to fill in the title.

The End

And we’re done. Or we would be, but I would really prefer the header to read “March” instead of “2015-03 March”, and “Saturday 7 March 2015” instead of “2015-03-07 Saturday”. We might need to come back to that.

Emacs, Rvm, and the Important Distinction between Interactive and Non-Interactive Shells

Normally I run the non-graphical version of Emacs in iTerm and switch to a non-Emacs tab to run rspecs and cucumbers at the command line. The other day, since I have feature-mode installed, I typed C-c , v in a feature file to run the cucumber features. The start of the compilation buffer of the results showed:

-*- mode: compilation; default-directory: "~/code/my-project/features/" -*-
Compilation started at Sat Feb 28 00:24:30

rake cucumber CUCUMBER_OPTS="" FEATURE="/Users/seanmiller/code/my-project/features/smoke.feature"
(in /Users/seanmiller/code/my-project)
rake aborted!
undefined method `result' for #<TypeError:0x007fa8ca0f51c0>
/Users/seanmiller/.rvm/gems/ruby-1.9.3-p551@my-project/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `translate_exception'

with a long stack trace mostly in the activerecord adapters.

The third line (starting “rake cucumber…”) is the command that was run, so we can type M-x eshell to get a shell within Emacs, paste in the line and run it, and there the cucumbers run without error. We can also type M-x compile to get a compile command prompt directly, paste in the line again, and we end up with the error we started with. The compile command prompt and the Emacs shell are behaving differently.

What if we run echo $PATH in both? The shell path starts with several rvm-related directories. The compile prompt path starts with /usr/bin and the rvm-related directories are nearer the end.

That might well do it, but what is causing it? The manual for the compilation shell notes that it brings up a noninteractive shell. The manual for zsh notes that while ~/.zshenv is run for almost every zsh, ~/.zshrc is run only for interactive shells. So if I get the latest version of the rvm dotfiles via rvm get stable --auto-dotfiles and move the two lines that show up in ~/.profile not to my ~/.zshrc file but to my ~/.zshenv file, and then restart Emacs, I can run the cukes from M-x compile or C-c , v and they run cleanly.

What’s more, in the compilation buffer of the test results, the comments referring to the line numbers of feature and step definition files are live links: press enter on them to jump to the lines of the code files. Maybe there is a point to running them in Emacs after all…

Shortcuts to Default Directories

[Now largely historical. See ETA for achieving the same effect with bookmarks, and ETA 2 for achieving the same effect with Hydra, which feels better still.]

Yesterday’s Emacs coaching session with Sacha Chua included a sidebar on jumping to and setting default directories. Sacha’s .emacs.d uses registers for this, and her code sets defaults and org-refile targets.

I found later that I had six shortcuts I’d clean forgotten about, that set the default directory and open either dired or a specific file:

(global-set-key (kbd "C-c C-g C-c")
(lambda ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired ".")))

(global-set-key (kbd "C-c C-g C-h")
(lambda ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org")))

But a forgotten solution doesn’t count, and having to remember triple-decker key bindings doesn’t help either.

A general solution for the second problem is that an incomplete key binding followed by ? opens a help buffer with all the bindings starting with sequence. C-c C-g ? on my machine yielded:

Global Bindings Starting With C-c C-g:
key binding
— ——-

C-c C-g C-c ??
C-c C-g C-d ??
C-c C-g C-e to-english
C-c C-g C-f to-french
C-c C-g C-h ??
C-c C-g C-p ??
C-c C-g C-r ??
C-c C-g C-u ??
C-c C-g C-w ??

We can make this more useful by switching from defining the key binding functions anonymously inline to moving them out to their own named functions, thus:

(defun my/to-emacs ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired "."))

(defun my/to-today ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org"))

(global-set-key (kbd "C-c C-g C-c") ‘my/to-emacs)
(global-set-key (kbd "C-c C-g C-h") ‘my/to-today)

Which fixes those two ?? entries:

C-c C-g C-c my/to-emacs
C-c C-g C-h my/to-today

Another niggle is that I’ve overloaded C-c C-g across three different kinds of commands. Let’s make C-x j the shortcut specifically and only for jumping to a new default directory, and extract some duplicate methods while we’re at it. After:

(defun my/to-file (dir file)
(interactive)
(setq default-directory dir)
(find-file file))

(defun my/to-dir (dir)
(interactive)
(setq default-directory dir)
(dired "."))

(defun my/to-gesta-file (file)
(interactive)
(my/to-file "~/Dropbox/gesta/" file))

(defun my/to-emacs-config ()
(interactive)
(my/to-file "~/.emacs.d/" "sean.org"))

(defun my/to-autrui ()
(interactive)
(my/to-dir "~/code/autrui/"))

(defun my/to-gesta ()
(interactive)
(my/to-dir "~/Dropbox/gesta/"))

(defun my/to-today ()
(interactive)
(my/to-gesta-file "2015.org"))

(defun my/to-readings ()
(interactive)
(my/to-gesta-file "readings.org"))

(defun my/to-writings ()
(interactive)
(my/to-gesta-file "writings.org"))

(defun my/to-twc ()
(interactive)
(my/to-dir "~/Dropbox/gesta/twc/"))

(global-set-key (kbd "C-x j e") ‘my/to-emacs-config)
(global-set-key (kbd "C-x j a") ‘my/to-autrui)
(global-set-key (kbd "C-x j g") ‘my/to-gesta)
(global-set-key (kbd "C-x j h") ‘my/to-today)
(global-set-key (kbd "C-x j r") ‘my/to-readings)
(global-set-key (kbd "C-x j w") ‘my/to-writings)
(global-set-key (kbd "C-x j t") ‘my/to-twc)

C-x j ? then yields:

Global Bindings Starting With C-x j:
key binding
— ——-

C-x j a my/to-autrui
C-x j e my/to-emacs-config
C-x j g my/to-gesta
C-x j h my/to-today
C-x j r my/to-readings
C-x j t my/to-twc
C-x j w my/to-writings

Which is an improvement, but the “Bindings Starting With” help menu is not interactive: we still have to either remember the triple-decker key binding, or type C-x j ? to find the list and then type C-x j h (say) to get the specific shortcut we want. It would be nice if we could over-ride C-x j ? and have it prompt us for one more character to select which of the seven jumps we wanted.

Something like this, in fact:

(defun my/pick-destination (pick)
(interactive "ce = ~/.emacs.d/sean.org a = ~/code/autrui/ g = ~/Dropbox/gesta/ h = …/2015.org r = …/readings.org w = …/writings.org t = …/twc/ ? ")
(case pick
(?e (my/to-emacs-config))
(?a (my/to-autrui))
(?g (my/to-gesta))
(?h (my/to-today))
(?r (my/to-readings))
(?w (my/to-writings))
(?t (my/to-twc))))

(global-set-key (kbd "C-x j ?") ‘my/pick-destination)

interactive "c… takes a single character (see further options here), so the conditional is on a character, which in Emacs Lisp is represented by a ? followed by the character {n}. If there were a single conditional we might use (when (char-equal ?e pick) {n}, but since there are seven of them, we look instead for Emacs Lisp’s equivalent of a switch statement, and find it in case (an alias for cl-case).

So what this does, when you type C-x j ?, is provide a prompt of options in the echo area at the bottom of the screen, and if you type one of the significant letters, it uses that shortcut to default.

Purists would probably prefer to bind ’my/pick-destination to something other than C-x j ? (C-x j j say), so that if we ever bound other commands to something starting with C-x j we would still be able to discover them with C-x j ?. It’s also easier to type because it doesn’t need the shift key for the third element. Having demonstrated that we could over-ride C-x j ? if we wanted to, I’m probably going to side with the purists on this one.

And we’re done. Commit.

ETA: Easier Ways To Do It

That works, but since (see Sacha’s comment below) there are always easier ways to do it:

Simpler Code

find-file will either open a file or open dired if it is passed a directory instead of a file, and opening a file or directory will change the default directory. So we can replace all the extracted ’my/to-file and ’my/to-dir methods with simple find-file calls:

(defun my/to-emacs-config ()
(interactive)
(find-file "~/.emacs.d/sean.org"))

(defun my/to-autrui ()
(interactive)
(find-file "~/code/autrui/"))

… etc …

Using Bookmarks (No Custom Code, Even Simpler)

While ’my/pick-destination gets around the non-interactive nature of ? (minibuffer-completion-help, e.g. C-c C-g ? for list of completions to C-c C-g), using straight-up bookmarks lets you do that without custom code:

C-x r m {name} RET ;; bookmark-set ;; sets a {named} bookmark at current location
C-x r b {name} RET ;; bookmark-jump ;; jump to {named} bookmark
C-x r l ;; list-bookmarks ;; list all bookmarks

So having set bookmarks in the same seven places, either files or directories, we could list-bookmarks and get

% Bookmark File
autrui ~/code/autrui/
emacs ~/.emacs.d/sean.org
gesta ~/Dropbox/gesta/
now ~/Dropbox/gesta/2015.org
readings ~/Dropbox/gesta/readings.org
twc ~/Dropbox/gesta/twc/
writings ~/Dropbox/gesta/writings.org

And this list is interactive. Or we could call C-x r b (bookmark-jump) and start typing the destination we want. (If we get bored of typing, we can reduce the bookmark names to single characters, remembering that C-x r l (list-bookmarks) will give us the translation table if we forget, and then we’re back to being five keystrokes away from any bookmark, without having to add any extra code.)

Using Bookmark+ (And Describing and Customizing Faces)

If we want to be able to do more advanced things with bookmarks – tag them, annotate them, rename them, run dired-like commands on the bookmark list – we can grab the Bookmark+ package and (require ’bookmark+) in our .emacs. (Having done that, if we press e by a line in the bookmark list, for instance, we can edit the lisp record for the bookmark, to rename it or change the destination or see how many times it has been used.)

One problem I had with bookmark+ is that the bookmark list was displaying illegibly in dark blue on a black background. To fix this, I needed to move the cursor over one of the dark blue on black bookmark names and type M-x describe-face, and it reported the face (Emacs-speak for style {n}) of the character, in this case Describe face (default `bmkp-local-file-without-region'):. Pressing enter took me to a buffer which described the face and provided a customize this face link at the end of the first line. I followed (pressed enter) that link to get to a customize face buffer which let me change the styling for that element. On the line:

[X] Foreground: blue [ Choose ] (sample)

I followed (pressed enter on) Choose, it opened a buffer of colour options, I scrolled up to a more visible one, pressed enter again, and got back to the customize face buffer, then went up to “Apply and Save” link and pressed enter again there. Going back to the bookmark list, the bookmarks were visible. The change is preserved for future sessions in the init.el file:

(custom-set-faces
;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won’t work right.
‘(bmkp-local-file-without-region ((t (:foreground "green")))))

ETA 2: Using Hydra Instead

Heikki Lehvaslaiho suggested using hydra instead, and as the bookmark solution required a couple of extra keystrokes and I’d been curious about hydra anyway, I thought I’d give it a go.

(require ‘hydra)
(global-set-key
(kbd "C-c j")
(defhydra hydra-jump (:color blue)
"jump"
("e" (find-file "~/.emacs.d/sean.org") ".emacs.d")
("c" (find-file "~/.emacs.d/Cask") "Cask")

("a" (find-file "~/code/autrui/") "autrui")
("h" (find-file "~/Dropbox/gesta/2015.org") "hodie")
("r" (find-file "~/Dropbox/gesta/readings.org") "readings")
("w" (find-file "~/Dropbox/gesta/writings.org") "writings")
("t" (find-file "~/Dropbox/gesta/twc/") "twc")))

I like it. We’re back down to three keystrokes from five and, since we’re providing labels for the commands, we also get the descriptive and interactive and self-updating index which was the big advantage of the bookmark route. If you press C-c j and don’t immediately carry on, it prompts:

jump: [e]: .emacs.d, : Cask, [a]: autrui, [h]: hodie, [r]: readings, [w]: writings, [t]: twc.

Much better. Thanks for the prompt, Heikki!

Literate Emacs Configuration

What?

Literate programming, proposed by Donald Knuth in 1984 {n}, suggests that

Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Literate Emacs configuration? Lets us build our .emacs.d files in .org files instead of .el files, so that with very little extra effort they can look like this.

In case that doesn’t speak for itself:

  • with org-mode’s headers and subheaders, you can quickly find your way around a large file, hiding all but the subheader content you care about
  • with org-mode’s links, the links to the snippets from What the .emacs.d!? or Avdi’s Emacs Reboot are live links
  • with org-mode’s export to html, you can weave an even more readable version of the config.
  • since it’s org-mode, you can put TODO statuses or tags on headers as well, so you can easily record the status of your practice on some shortcuts, or tag parts of the config as :experimental:.

How?

The simplest version really does work out of the box.

In your init.el file, include an org-babel-load-file reference to the org file you’re going to use, in my case ~/.emacs.d/sean.org:

(org-babel-load-file "~/.emacs.d/sean.org")

Then, create the org file, and put whatever bits of config that you want in emacs-lisp source blocks, like so:

#+BEGIN_SRC emacs-lisp
  ... code here ...
#+END_SRC

What this will do is export the source code using org-babel-tangle (more on that later) into a file ~/.emacs.d/sean.el, and it will then load the resulting file using load-file. It is effectively the same as

(load-file "~/.emacs.d/sean.el")

except that it checks the timestamps of the sean.el and sean.org files and if the sean.org file was changed later, it re-extracts the sean.el file.

(Since sean.el is a generated file you will probably want to add it to your project’s .gitignore so it isn’t committed.)

And in the sean.org file you can put headings and subheadings to make it easy to navigate: in my first try, because my Emacs config had previously been across several files, I used the file names as subheadings, ending up with something like this:

* Emacs Config
** Defaults…
** Mode hooks…
** Global key bindings…
** Project-specific shortcuts…
** Zenburn theme…
** Emacs server and Emacsclient…
…etc

Check the Html Output

It is the case that if you export that from org to html right now (C-c C-e h o / org-export-dispatch) the h1 title will be the filename, which is unhelpfully just “sean”. Give it a better name with an explicit title export setting at the top of the file:

#+TITLE: Sean Miller’s Emacs Configuration

Re-export, and the html file has that title, and, by default, a linked table of contents derived from the headers and subheaders. (Other export options are described here.)

Run the Tests: A Catch

The next thing that I did was try to run the tests (tests introduced here, test runner introduced here), after changing the initial load-file line in the test to point to the generated sean.el file:

(load-file "sean.el")

And… I got an abnormal exit from ert-runner because the server-start from another part of my Emacs config prevented ert-runner from starting a server or running any of the tests.

We can fix that by breaking apart the emacs-lisp blocks from sean.org into two separate files, one of settings and one of code-under-test, and then in the test we can load the code-under-test file on its own.

To do this, we add a :tangle argument to the #+BEGIN_SRC emacs-lisp blocks in the sean.org file, with a filename argument. Code that is not under test gets the begin line:

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-settings.el

And code under test gets the begin line

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-code.el

Then, when we make changes to the sean.org file, on save we can run the org-babel-tangle command and it will export the emacs-lisp blocks into ~/.emacs.d/tangled-settings.el and ~/.emacs.d/tangled-code.el respectively.

We can then remove the org-babel-load-file line from init.el and replace it with:

(load-file "~/.emacs.d/tangled-settings.el")
(load-file "~/.emacs.d/tangled-code.el")

So that the next time we restart Emacs it will pick up the two built files, and if we change the load in our test file to be

(load-file "tangled-code.el")

And run our tests again, this time they run fine, so we have literate Emacs config and running tests.

(You’ll want to change your project’s .gitignore again to include the new “tangled-” files.)

Why “tangled”? It comes from the terminology of literate programming: the literate source file can be “tangled” to produce machine-readable code (exporting the source code blocks into .el files), and “woven” to produce formatted documentation. Knuth later admitted {n} that the echo of Walter Scott’s “Oh, what a tangled web we weave when first we practise to deceive” was entirely deliberate.

Automating the Manual Step We Just Introduced

When we fixed the catch we lost the automatic updating of the .el files when we changed the .org file. Now we need to remember to run org-babel-tangle manually after changing the .org file.

Manual steps are invariably forgotten, so let’s automate. The simplest hook would be:

(add-hook ‘after-save-hook ‘org-babel-tangle)

But we only want to tangle if we’ve just saved the Emacs config org file, so let’s build a new function to check that before running tangle, and hook to that instead:

(defun my/tangle-on-save-emacs-config-org-file()
(when (string= buffer-file-name (file-truename "~/.emacs.d/sean.org"))
(org-babel-tangle)))

(add-hook ‘after-save-hook ‘my/tangle-on-save-emacs-config-org-file)

buffer-file-name returns the absolute path of the name of the file the buffer is visiting. file-truename converts ~/.emacs.d/sean.org into an absolute path so that the string comparison works.

And with that, we’re done. Commit.

Trailing Whitespace in Commits

The Problem

You’re opening a pull request to do a code review. There are 34 changed files. You find this distressing, but you dig in and find only three changed files contain changes in logic: the rest are removing whitespace. You examine the three easily and go on to what’s next.

But suppose you had seen there were 34 changed files and cringed and gone on to another task instead? It would be better if there were a way to quickly distinguish real changes from formatting changes. Better still if we could prevent commits that add whitespace we’re only going to take out later in the first place.

Hiding but Not Solving the Problem

git diff, of course, shows you the differences between two code trees.

git diff -w / git diff --ignore-all-space shows you the differences excluding whitespace differences.

You can get github to ignore whitespace in its diffs by appending ?w=1 to the URL (say, of the pull request that you’re reviewing).

This solves the cognitive load problem – being able to tell at a glance that 31 of the 34 changed files are not significant is a huge plus – but all of those extra files have still been changed. Given a small team of great developers, no problem, but if you’ve got multiple people at multiple skill levels working on multiple branches, merges become more complicated and merge conflicts more likely than they need to be.

Solving the Problem: Going Forward

Any editor any developer on your team is likely to be using (Emacs, Vim, RubyMine, Sublime Text, TextMate) already has the ability to automatically remove trailing whitespace before saving a file. Make sure they all have it switched on. For Emacs, just add this to your config:

(add-hook ‘before-save-hook ‘delete-trailing-whitespace)

For other editors, there are instructions here.

Solving the Problem: Everything before That

If the project already has three years of commits before you switched that on, there will still be a problem with already-committed trailing whitespace.

What I would do in that case is find the least disruptive moment, grab something like rstrip (which is more flexible than one-liners from the command line because it starts by setting up a config file so you can decide which file extensions are processed), run it on the whole project, and commit the result. This will make you very unpopular with people on outlying branches, hence the importance of finding the least disruptive moment, but after that, you’ve fixed the trailing whitespace problem for good.

Teaser for Another Problem

Trailing whitespace is a single instance of the larger problem of automatically enforcing coding style conventions.

Back in Java this was a solved problem: write code with Eclipse, use the Checkstyle and Jalopy plugins (or their IntelliJ equivalents): Checkstyle creates warnings whenever a coding style convention is breached, and can also stop you committing code before resolving those warnings. Invoking Jalopy on Checkstyled code can automatically reformat it to follow the Checkstyle conventions to make it committable.

In Ruby? Thoughtbot has released hound which automatically comments on pull requests with Ruby / Javascript / Coffeescript / SCSS style coding style violations, based on Thoughtbot’s style guides.

Bozhidar Batsov (bbatsov) has a Ruby static code analyzer called rubocop which can be run locally and will produce warnings based on his community-driven Ruby style guide.

Yorick Peterse has another Ruby static code analyzer called ruby-lint.

Investigating all these is a large enough topic for another post.