Testing with Page Objects: Implementation

In the previous post, we added the page object framework SitePrism to a project, did some organization and customization, and added helper methods so we could call visit_page and on_page to interact with the app. Next, let’s start using it.


First Step

Let’s say that for the first test you want to go to the login page, log in, and then assert that you’re on the dashboard page. That’s easy enough to do like so:

visit_page("Login").login(username, password)

module PageObjects
  class Login < SitePrism::Page
    set_url "/login"

    element :username, "input#login"
    element :password, "input#password"
    element :login, "input[name='commit']"

    def login(username, password)
      self.username.set username
      self.password.set password

module PageObjects
  class Dashboard < SitePrism::Page
    set_url "/dashboard"

The first line will go to the “/login” page and then try to submit the form, and will fail if it can’t find or interact with the “username”, “password”, or “login” elements. For instance, if the css for the username field changes, the test will fail with:

Unable to find css "input#login" (Capybara::ElementNotFound)
./test_commons/page_objects/login.rb:10:in `login'
./features/step_definitions/smoke.rb:7:in `login'
./features/step_definitions/smoke.rb:3:in `/^I log in$/'
features/smoke.feature:5:in `When I log in’

The second line will fail if the path it arrives at after logging in is something other than “/dashboard”.

Because the default message for the different path was uninformative (expected true, but was false), we added a custom matcher to provide more information. The default message for not finding an element on the page, including the css it’s expecting and the backtrace to the page object it came from and the line in the method it was trying to run, seems sufficiently informative to leave as-is.

If the login field were disabled instead of not found, the error message would be slightly less informative:

invalid element state
  (Session info: chrome=…)
  (Driver info: chromedriver=2.14. … (Selenium::WebDriver::Error::InvalidElementStateError)
./test_commons/page_objects/login.rb:10:in `login'
./features/step_definitions/smoke.rb:7:in `login'
./features/step_definitions/smoke.rb:3:in `/^I log in$/'
features/smoke.feature:5:in `When I log in’

but since the backtrace still gives you the line of the page object where it failed, and that gets you the element in an invalid state, it feels lower priority.

Second Step

If the next thing you want to test is a subcomponent flow, you can navigate to its first page either by calling visit_page('ComponentOne::StepOne') (the equivalent of typing the path in the browser address bar), or by having the test interact with the same navigation UI that a user would. The second choice feels more realistic and more likely to expose bugs, like the right navigation links not being displayed.

(Ironically, having created the visit_page method, we might only ever use it for a test’s first contact with the app, using in-app navigation and on_page verification for all subsequent interactions.)

To get that to work, we need to model the navigation UI.

SitePrism Sections and Navigation UI

Supose the app has a navigation bar on every logged-in page. We could add it as a section to the Dashboard, say, but if it’s on every page it probably makes more sense to add it as a section to a BasePage and have Dashboard (and all other logged-in pages) inherit from it:

module PageObjects
  class BasePage < SitePrism::Page
    section :navigation, Navigation, ".navigation"

module PageObjects
  class Dashboard < BasePage

module PageObjects
  class Navigation < SitePrism::Section
    element :component_one, ".component_one"
    element :componont_two, ".component_two"

    def goto_component_one

    def goto_component_two

Now, if we want to go from the dashboard to the start of component one, we can modify the on_page("Dashboard") line above to


Can we delegate to the section from the enclosing page? Yes. If we add this to the base page:

module PageObjects
  class BasePage < SitePrism::Page
    delegate :goto_component_one, :goto_component_two, to: :navigation

we can then call, still more briefly:


And now the test from the top can continue like so:

visit_page("Login").login(username, password)

and line two will fail if the link to “component one” isn’t visible on the screen, and line three will fail if the user doesn’t end up on the first page of component one (perhaps they followed the link but authorizations aren’t set up correctly, so they end up on an “unauthorized” page instead).

What Else Is on the Page?

If you call a method on a page object it will verify that any item that it interacts with is on the page, and fail if it can’t find it or can’t interact with it. But what if you’re adding tests to an existing app, and you aren’t yet calling methods that interact with everything on the page, but you do want to make sure certain things are on the page?

Suppose you have a navigation section with five elements, unimaginatively called :one, :two, :three, :four, :five. And you have three types / authorization levels of users, like so:

User Can see elements
user1 :one
user3 :one, :two, :three
user5 :one, :two, :three, :four, :five

And you want to log each of them in and verify that they only see what they should see. We can do this with SitePrism’s has_<element>? and has_no_<element>? methods.

A naive approach might be:

all_navigation_elements = %w{one two three four five}

visit_page('Login').login(user1.username, user1.password)
on_page('Dashboard') do |page|
  user1_should_see = %w{one}
  user1_should_see.each do |element|
    expect(page.navigation).to send("have_#{element}")
  user1_should_not_see = all_navigation_elements - user1_should_see
  user1_should_not_see.each do |element|
    expect(page.navigation).to send("have_no_#{element}")

Even before doing the other two, it’s clear that we’re going to want to extract this into a more generalized form.

To build out the design by “programming by wishful thinking”, if we already had the implementation we might call it like this:

all_navigation_elements = %w{one two three four five}

visit_page('Login').login(user1.username, user1.password)
on_page('Dashboard') do |page|
  should_see = %w{one}
  has_these_elements(page.navigation, should_see, all_elements)

visit_page('Login').login(user3.username, user3.password)
on_page('Dashboard') do |page|
  should_see = %w{one two three}
  has_these_elements(page.navigation, should_see, all_elements)

# do. for user5

And then we can reopen test_commons/page_objects/helpers.rb to add an implementation:

def has_these_elements(page, has_these, all)
  has_these.each do |element|
    expect(page).to send("have_#{element}")
  does_not_have = all - has_these
  does_not_have.each do |element|
    expect(page).to send("have_no_#{element}")

And this works. The raw error message (suppose we’re doing this test first so we’ve got the test for user1 but user1 still sees all five items) is unhelpful:

expected #has_no_two? to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:28:in `block in has_these_elements'
./test_commons/page_objects/helpers.rb:27:in `each'
./test_commons/page_objects/helpers.rb:27:in `has_these_elements'
./features/step_definitions/navigation_permissions.rb:24:in `block in dashboard'

And we could build a custom matcher for it, but in this case since the error is mostly likely to show up (if we’re working in Cucumber) after

When user1 logs in
Then user1 should only see its subset of the navigation

seeing expected #has_no_two? after that seems perfectly understandable.

Summing Up So Far

I’m quite liking this so far, and I can easily imagine after I’ve used them for a bit longer bundling the helpers into a gem to make it trivially easy to use them on other projects. And possibly submitting a PR to suggest adding something like page.has_these_elements(subset, superset) into SitePrism, because that feels like a cleaner interface than has_these_elements(page, subset, superset).

Testing with Page Objects: Setup

What Are Page Objects?

Page objects are an organizing tool for UI test suites. They provide a place to identify the route to and elements on a page and to add methods to encapsulate paths through a page. Martin Fowler has a more formal description.

Suppose you’re testing a web app which has a component with a four-page flow, with a known happy path a user would normally work through. Using page objects, you could test that flow as:

def component_one_flow(user, interests, contacts)

If anything changes on one of those pages, you can make the change inside the page object, and the calling code can remain the same.

For small- to medium-sized web apps this might seem a nice-to-have, but for larger apps where you end up afraid to open features/step_definitions or spec/features because it’s impossible to find anything, this is a very useful pattern to introduce, or to start out with on the next project.

This ran quite long, so I’ve split it into two parts: in this first post I’ll describe setting up a page object framework, and in the second I’ll talk about implementation patterns.

Available Frameworks

In 2013 and 2014 I enjoyed working with Jeff Morgan’s page-object, which runs on watir-webdriver. (His book Cucumber and Cheese was my introduction to page objects.) Jeff Nyman’s Symbiont also runs on watir-webdriver and also looks quite interesting (as does his testing blog). There’s also Nat Ritmeyer’s SitePrism, running on Capybara.

Here’s a login page in all three:


class LoginPage
  include PageObject

  page_url "/login"

  text_field(:username, id: 'username')
  text_field(:password, id: 'password')
  button(:login, id: 'login')

  def login_with(username, password)
    self.username = username
    self.password = password


class Login < SitePrism::Page
  set_url "/login"

  element :username, "input#login"
  element :password, "input#password"
  element :login, "input[name='commit']"

  def login_with(username, password)
    self.username.set username
    self.password.set password


class Login
  attach Symbiont

  url_is      "/login"
  title_is    "Login"

  text_field :username, id: 'user'
  text_field :calendar, id: 'password'
  button     :login,    id: 'submit'

  def login_with(username, password)
    self.username.set username
    self.password.set password

The similarities are clear: in each case you can identify the URL, the elements, and methods that interact with those elements. And since it is the internal page object names for elements that are used in the methods, if (when) the css changes, the only line that needs to change in the page object is the declaration of the element.

What about differences? Page-object (here) and Symbiont (here) provide page factories, so that in page-object you can write

visit(LoginPage) do |page|
  page.login(username, password)

or, even more compactly

visit(LoginPage).login(username, password)

where SitePrism is, somewhat more long-windedly:

page = Login.new
page.login(username, password)

SitePrism offers, in addition to element, the named concepts of elements (for a collection of rows, say) and section (for a subset of elements on a page, or elements which are common across multiple pages, a navigation bar, say, which you can identify separately in a SitePrism::Section object).

The further point that inclined me to try SitePrism (though with a mental note to add a page factory) was that when we were using page-object last year we sometimes needed to write explicit wrappers for waiting code because the implicit when_present wait did not always seem to be reliable, and it looked as though Symbiont wrapped it the same way (here). I was curious to see if SitePrism (or its underlying driver) had a more robust solution.

Setting up SitePrism

So How Does It Handle Waiting?

By default, SitePrism requires explicit waits, so either wait_for_<element> or wait_until_<element>_visible. This has been challenged (issue#41, in which Ritmeyer explains that he likes having more fine-grained control over timing of objects than Capybara’s implicit waits allows), and a compromise has been implemented (issue#43), by which you can configure SitePrism to use Capybara’s implicit waits, so:

SitePrism.configure do |config|
  config.use_implicit_waits = true

I’ve been using that, and it’s working so far.

Where to Put Them

On the earlier project we were using page objects from Cucumber, so we put them under features/page_objects. The new project is larger and already has non-test-framework-specific code under both features and spec, some of which is called from the other framework’s directory, so it makes sense to create a common top-level directory for them. I called it test_commons to make it clear at a glance through the top-level directories that it was test-related instead of app-related. Then page_objects goes under that.

The disadvantage of the location is that we will need to load the files in explicitly. So we add this at test_commons/all_page_objects.rb (approach and snippet both from Panayotis Matsinopoulos’s useful post):

page_objects_path = File.expand_path('../page_objects', __FILE__)
Dir.glob(File.join(page_objects_path, '**', '*.rb')).each do |f|
  require f

and in features/support/env.rb we add:

require_relative '../../test_commons/all_page_objects'

and we’re away. (If/when we use them from RSpec features as well, we can require all_page_objects in spec_helper.rb too.)

Further Organization

Probably you’ll have some top-level or common pages, like login or dashboard, and others which belong to specific flows and components, like the four pages of “component one” in the first example. A login page could go in the top-level page objects directory:

# test_commons/page_objects/login.rb
module PageObjects
  class Login < SitePrism::Page

whereas the four pages of component one could be further namespaced:

# test_commons/page_objects/component_one/step_one.rb
module PageObjects
  module ComponentOne
    class StepOne < SitePrism::Page

This becomes essential when you’ve got more than a handful of pages.

Adding a Page Factory for SitePrism

I started by looking at page-object’s implementation (which itself references Alister Scott’s earlier post).

That seemed to do more than I needed right away, so I started with the simpler:

# test_commons/page_objects/helpers.rb
module PageObjects
  def visit_page(page_name, &block)
    Object.const_get("PageObjects::#{page_name}").new.tap do |page|
      block.call page if block

… though we will need to add World(PageObjects) to our features/support/env.rb to use this from Cucumber (the require got us the page object classes, but not module methods). If we were running from RSpec, to borrow a snippet from Robbie Clutton and Matt Parker’s excellent LA Ruby Conference 2013 talk, we would need:

RSpec.configure do |c|
  c.include PageObjects, type: [:feature, :request]

So now we can call visit_page (not visit because that conflicts with a Capybara method), it’ll load the page (using the page object’s set_url value), and if we have passed in a block it’ll yield to the block, and either way it’ll return the page, so we can call methods on the returned page. In other words: visit_page('Login').login(username, password).

We can use the same pattern to build an on_page method, which instead of loading the page asserts that the app is on the page that the test claims it will be on:

def on_page(name, args = {}, &block)
  Object.const_get("PageObjects::#{page_name}").new.tap do |page|
    expect(page).to be_displayed
    block.call page if block

Further iteration (including the fact that both page.load and page.displayed? take arguments for things like ids in URLs) results in something like this:

module PageObjects
  def visit_page(name, args = {}, &block)
    build_page(name).tap do |page|
      block.call page if block

  def on_page(name, args = {}, &block)
    build_page(name).tap do |page|
      expect(page).to be_displayed(args)
      block.call page if block

  def build_page(name)
    name = name.to_s.camelize if name.is_a? Symbol

We haven’t implemented page-object’s if_page yet (don’t assert that you’re on a page and fail if you aren’t, but check if you’re on a page and if you are carry on), but we can add it later if we need it.

What Would that Failed Assertion Look Like?

Suppose we tried on_page('Dashboard').do_something when we weren’t on the dashboard. What error message would we get?

expected displayed? to return true, got false (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:11:in `block in on_page'

We know, because SitePrism has a #current_path method, what page we’re actually on. A custom matcher to produce a more informative message seems a good idea. Between the rspec-expectations docs and Daniel Chang’s exploration, let’s try this:

# spec/support/matchers/be_displayed.rb
RSpec::Matchers.define :be_displayed do |args|
  match do |actual|

  failure_message_for_should do |actual|
    expected = actual.class.to_s.sub(/PageObjects::/, '')
    expected += " (args: #{args})" if args.count > 0
    "expected to be on page '#{expected}', but was on #{actual.current_path}"

which we’d get for free from RSpec, and from Cucumber we can get by adding to features/support/env.rb:

require_relative '../../spec/support/matchers/be_displayed'

The first time I ran this, because I wasn’t passing in arguments to SitePrism’s #displayed? method, I hit a weird error:

comparison of Float with nil failed (ArgumentError)
./spec/support/matchers/be_displayed.rb:3:in `block (2 levels) in <top (required)>'

… emending the helper methods to make sure that I included an empty hash even if no arguments were passed in (part of the final version of helpers.rb above) fixed that. Now, with the new matcher working, if we run on_page('Dashboard').do_something when we’re actually at the path /secondary-control-room, we get the much more useful error

expected to be on page 'Dashboard', but was on '/secondary-control-room' (RSpec::Expectations::ExpectationNotMetError)
./test_commons/page_objects/helpers.rb:11:in `block in on_page'

… note that if there had been expected arguments, we would have printed those out too (expected to be on page 'Dashboard' (args: {id: 42})).

I built the custom matcher in RSpec because that’s what I tend to use, but if Minitest is more your thing, you could add a custom matcher by reopening Minitest::Assertions. (Example here.)

Setup Complete

We’ll break now that the page object framework is set up, and discuss further implementation patterns in the next post.

Emacs Org-mode Styling, Non-smart Quotes, Zero-width-space, and TeX Input Method

In org-mode we can style inline elements with *bold* (bold), /italic/ (italic), _underlined_ (_underlined_, which won’t show up in html), =verbatim= (verbatim), and ~code~ (code).

But this breaks if the character just inside the styling code is a non-smart single or double quote. =C-c ;= is styled (C-c ;); =C-c '= is not (=C-c '=).

We can fix that by inserting a zero-width space between the apostrophe and the = . The first time, we can put the cursor between the apostrophe and the = and enter C-x 8 RET ZERO WIDTH SPACE RET, at which point =C-c '​= will display as C-c '.

The ZERO WIDTH SPACE part of insert-char (C-x 8) is autocompleting, so we won’t need to type the whole thing, but we’ll probably still want a shortcut if it we find ourselves doing it more than once or twice. Here’s the simplest thing we could do:

(defun my/insert-zero-width-space ()
(insert-char ?\u200B)) ;; code for ZERO WIDTH SPACE
(global-set-key (kbd "C-x 8 s") ‘my/insert-zero-width-space)

After which, if we find ourselves adding inline styles which don’t work because the styled section starts or ends with a quotation mark, we can go to just before or just after the quotation mark, type C-x 8 s, and watch the styles apply as expected.

Thanks, as usual, to Sacha Chua, who explained this in passing. Her .emacs has a more flexible solution which I will probably switch to the moment I need to insert more than one unicode character.

Note that while you can use insert-char to get accented characters (C-x 8 ' e produces é, for instance), and I used to do that, I have since set the input method to TeX (M-x set-input-method RET TeX RET), and now I type \​'e for é, \​th for þ, \​Mu\​epsilon\​nu for Μεν, and so on and so forth.

Automating Boilerplate in Org-mode Journalling

The Problem

My main org-mode files have a header structure like this:

* 2015
** March
*** Friday 6 March 2015
**** 10:00. Emacs coaching with Sacha :emacs:
<2015-03-06 10:00-11:00>
**** 23:37. Emacs progress note
[2015-03-06 23:37]
*** Saturday 7 March 2015
**** 12:55. Stratford King Lear in cinemas :wcss:
<2015-03-07 12:55-15:55>

The two things I do most often are start writing notes in the present, and set up scheduled events in the future. Scheduled events need active timestamps so that they show up in the agenda view:

Week-agenda (W10):
Monday 2 March 2015 W10
Tuesday 3 March 2015
Wednesday 4 March 2015
Thursday 5 March 2015
Friday 6 March 2015
10:00. Emacs Coaching with Sacha :emacs:
Saturday 7 March 2015
12:55. Stratford King Lear in cinemas :wcss:
Sunday 8 March 2015

[How to get the agenda view? Make sure the file you’re in is in org-agenda-files by pressing C-c [, then run org-agenda either by M-x org-agenda or the common (but not out-of-the-box) shortcut C-c a, then press a to get the agenda for the week.]

[Active timestamps are in < >; inactive timestamps are in [ ]. If we want to promote an inactive timestamp to be active and show up in the agenda view, we can run M-x org-toggle-timestamp-type from the inactive timestamp.]

[Start time is duplicated between header and timestamp deliberately: sometimes I review the file in org-mode outline with only the headers. If not for that use-case it would make sense to leave the start time out of the header.]

What I’d like to be able to do is open the file, and then with as little ceremony or setup as possible, start typing.

Adding a New Note

When I’ve already got a date header for the current date, this is trivially simple: I can put my cursor under that date and call M-x my note, having previously defined

(defun my/note ()
(insert "\n\n**** " (format-time-string "%H:%M") ". ")
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n")))

… and it does pretty much what it says.

The values passed into format-time-string (%Y, %m, %d etc.) will be the values for the current date and time.

save-excursion means “do what’s inside here and then return the point (cursor) to where it was before the block was called”, so it fills in the timestamp and then returns the cursor to the end of the header line, ready for the title to be filled in.

Since after that I’d need to move the cursor back down to below the timestamp, I can be even lazier and prompt for the title and the tags (if any) and fill them in and move the cursor down to where the text should start:

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

If we were just setting one argument, say title, this would be simpler:

(defun my/note-2-eg (title)
(interactive "MTitle? ")

But since we want two, we need to tell interactive it’s getting a list, and then the arguments are set in order.

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))

Just as we can create an inactive timestamp by inserting a formatted date with [ ], we can add tags by inserting a string surrounded by : :. We don’t want to do this if the tag argument was left blank, of course, so we surround it with an unless:

(unless (string= tags "")
(insert " :" tags ":")

[Side-note: if you’re doing anything more complicated than this with strings, you probably want Magnar Sveen’s s.el (string manipulation library), which here would let you do (unless (s-blank? tags)… instead.]

At the end of which, because we took out the save-excursion, the cursor is below the timestamp and I’m ready to type.

All good. What if we don’t already have a date header for the current date? Can we auto-generate the date header?

Auto-generating Date Headers

Org-mode has a function called org-datetree-find-date-create. If you pass it in a list of numbers (the month, the day, and the year), and if your header structure is

* 2015
** 2015-03 March
*** 2015-03-06 Friday

then if you called that function passing in (3 7 2015), it would automatically add:

*** 2015-03-07 Saturday

For that matter, if you called it passing in (4 1 2015), it would automatically add

** 2015-04 April
*** 2015-04-01 Wednesday

If you call it passing in a date which is already there, it moves the cursor to that date. So, we could change the format of the org file headers, update our new note function to call the function, and be done.

(defun my/note-3 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

In the new bit:

(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))

format-time-string “%Y-%m-%d” will return “2015-03-07”, and org-date-to-gregorian will turn that into (3 7 2015), which is the format that org-datetree-find-date-create expects. Determining this involved looking at the source for org-datetree-find-date-create to see what arguments it expected (C-h f org-datetree-find-date-create takes you to a help buffer that links to the source file, in this case org-datetree.el; click on the link to go to the function definition) and a certain amount of trial and error. At one point, before org-date-to-gregorian, I had the also working but rather less clear:

(org-datetree-find-date-create (mapcar ‘string-to-number
(split-string (format-time-string "%m %d %Y"))))

org-end-of-subtree just takes the cursor to the bottom of the section for the date.

And that then works. What about adding new events in the future?

Adding a New Event

org-datetree-find-date-create makes it easier to fill in missing month and date headers to create a new future event:

(defun my/event (date end-time)
(interactive (list
(read-from-minibuffer "end time (e.g. 22:00)? ")))
(org-datetree-find-date-create (org-date-to-gregorian date))
(goto-char (line-end-position))
(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")
(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n")))

There’s a problem I haven’t solved here, which is that org-read-date brings up the date prompt and lets you select a date, including a time or time range. If you select a time, it will be included in the date. If you select a time range, say 19:30-22:30, it ignores the time and the date object returned uses the current time. That’s not what we want.

So when the date prompt comes up:

Date+time [2015-03-07]: _ => <2015-03-07 Sat>

I can put in a new date and start time, say “2015-04-01 11:00”:

Date+time [2015-03-07]: 2015-04-01 11:00_ => <2015-04-01 Wed 11:00-14:00>

and then press enter and that gives me the date and the start time in a single string. We extract the start time and put it into the header like so:

(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")

Where split-string splits the date “2015-04-01 11:00” into a two-element list (“2015-04-01” “11:00”) {n}, and nth 1 returns the second element (list elements, here as elsewhere, are numbered starting from 0), or “11:00” {n}.

If the date had been “2015-04-01” without a start time, line 1 would have set start-time to nil, which would have blown up as an argument to insert (with a “Wrong type argument: char-or-string-p, nil”), so we check for a nil and set it to an empty string instead.

For the agenda view we’ll need the end-time as well, so we grab that as a second argument, and reassemble the date, say “2015-04-01 11:00”, and end-time, say “14:00”, into the active timestamp:

(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n"))

though here again, if we had just pressed enter when prompted for end time we would end up with an empty string and an invalid active timestamp, like “<2015-04-01->”, so check whether end-time has been set and build the timestamp string accordingly.

And since this is a future event and we’re probably only filling in the title, we’ll use save-excursion again to fill in the active timestamp and then go back and leave the cursor on the header to fill in the title.

The End

And we’re done. Or we would be, but I would really prefer the header to read “March” instead of “2015-03 March”, and “Saturday 7 March 2015” instead of “2015-03-07 Saturday”. We might need to come back to that.

Emacs, Rvm, and the Important Distinction between Interactive and Non-Interactive Shells

Normally I run the non-graphical version of Emacs in iTerm and switch to a non-Emacs tab to run rspecs and cucumbers at the command line. The other day, since I have feature-mode installed, I typed C-c , v in a feature file to run the cucumber features. The start of the compilation buffer of the results showed:

-*- mode: compilation; default-directory: "~/code/my-project/features/" -*-
Compilation started at Sat Feb 28 00:24:30

rake cucumber CUCUMBER_OPTS="" FEATURE="/Users/seanmiller/code/my-project/features/smoke.feature"
(in /Users/seanmiller/code/my-project)
rake aborted!
undefined method `result' for #<TypeError:0x007fa8ca0f51c0>
/Users/seanmiller/.rvm/gems/ruby-1.9.3-p551@my-project/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `translate_exception'

with a long stack trace mostly in the activerecord adapters.

The third line (starting “rake cucumber…”) is the command that was run, so we can type M-x eshell to get a shell within Emacs, paste in the line and run it, and there the cucumbers run without error. We can also type M-x compile to get a compile command prompt directly, paste in the line again, and we end up with the error we started with. The compile command prompt and the Emacs shell are behaving differently.

What if we run echo $PATH in both? The shell path starts with several rvm-related directories. The compile prompt path starts with /usr/bin and the rvm-related directories are nearer the end.

That might well do it, but what is causing it? The manual for the compilation shell notes that it brings up a noninteractive shell. The manual for zsh notes that while ~/.zshenv is run for almost every zsh, ~/.zshrc is run only for interactive shells. So if I get the latest version of the rvm dotfiles via rvm get stable --auto-dotfiles and move the two lines that show up in ~/.profile not to my ~/.zshrc file but to my ~/.zshenv file, and then restart Emacs, I can run the cukes from M-x compile or C-c , v and they run cleanly.

What’s more, in the compilation buffer of the test results, the comments referring to the line numbers of feature and step definition files are live links: press enter on them to jump to the lines of the code files. Maybe there is a point to running them in Emacs after all…

Shortcuts to Default Directories

[Now largely historical. See ETA for achieving the same effect with bookmarks, and ETA 2 for achieving the same effect with Hydra, which feels better still.]

Yesterday’s Emacs coaching session with Sacha Chua included a sidebar on jumping to and setting default directories. Sacha’s .emacs.d uses registers for this, and her code sets defaults and org-refile targets.

I found later that I had six shortcuts I’d clean forgotten about, that set the default directory and open either dired or a specific file:

(global-set-key (kbd "C-c C-g C-c")
(lambda ()
(setq default-directory "~/.emacs.d/")
(dired ".")))

(global-set-key (kbd "C-c C-g C-h")
(lambda ()
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org")))

But a forgotten solution doesn’t count, and having to remember triple-decker key bindings doesn’t help either.

A general solution for the second problem is that an incomplete key binding followed by ? opens a help buffer with all the bindings starting with sequence. C-c C-g ? on my machine yielded:

Global Bindings Starting With C-c C-g:
key binding
— ——-

C-c C-g C-c ??
C-c C-g C-d ??
C-c C-g C-e to-english
C-c C-g C-f to-french
C-c C-g C-h ??
C-c C-g C-p ??
C-c C-g C-r ??
C-c C-g C-u ??
C-c C-g C-w ??

We can make this more useful by switching from defining the key binding functions anonymously inline to moving them out to their own named functions, thus:

(defun my/to-emacs ()
(setq default-directory "~/.emacs.d/")
(dired "."))

(defun my/to-today ()
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org"))

(global-set-key (kbd "C-c C-g C-c") ‘my/to-emacs)
(global-set-key (kbd "C-c C-g C-h") ‘my/to-today)

Which fixes those two ?? entries:

C-c C-g C-c my/to-emacs
C-c C-g C-h my/to-today

Another niggle is that I’ve overloaded C-c C-g across three different kinds of commands. Let’s make C-x j the shortcut specifically and only for jumping to a new default directory, and extract some duplicate methods while we’re at it. After:

(defun my/to-file (dir file)
(setq default-directory dir)
(find-file file))

(defun my/to-dir (dir)
(setq default-directory dir)
(dired "."))

(defun my/to-gesta-file (file)
(my/to-file "~/Dropbox/gesta/" file))

(defun my/to-emacs-config ()
(my/to-file "~/.emacs.d/" "sean.org"))

(defun my/to-autrui ()
(my/to-dir "~/code/autrui/"))

(defun my/to-gesta ()
(my/to-dir "~/Dropbox/gesta/"))

(defun my/to-today ()
(my/to-gesta-file "2015.org"))

(defun my/to-readings ()
(my/to-gesta-file "readings.org"))

(defun my/to-writings ()
(my/to-gesta-file "writings.org"))

(defun my/to-twc ()
(my/to-dir "~/Dropbox/gesta/twc/"))

(global-set-key (kbd "C-x j e") ‘my/to-emacs-config)
(global-set-key (kbd "C-x j a") ‘my/to-autrui)
(global-set-key (kbd "C-x j g") ‘my/to-gesta)
(global-set-key (kbd "C-x j h") ‘my/to-today)
(global-set-key (kbd "C-x j r") ‘my/to-readings)
(global-set-key (kbd "C-x j w") ‘my/to-writings)
(global-set-key (kbd "C-x j t") ‘my/to-twc)

C-x j ? then yields:

Global Bindings Starting With C-x j:
key binding
— ——-

C-x j a my/to-autrui
C-x j e my/to-emacs-config
C-x j g my/to-gesta
C-x j h my/to-today
C-x j r my/to-readings
C-x j t my/to-twc
C-x j w my/to-writings

Which is an improvement, but the “Bindings Starting With” help menu is not interactive: we still have to either remember the triple-decker key binding, or type C-x j ? to find the list and then type C-x j h (say) to get the specific shortcut we want. It would be nice if we could over-ride C-x j ? and have it prompt us for one more character to select which of the seven jumps we wanted.

Something like this, in fact:

(defun my/pick-destination (pick)
(interactive "ce = ~/.emacs.d/sean.org a = ~/code/autrui/ g = ~/Dropbox/gesta/ h = …/2015.org r = …/readings.org w = …/writings.org t = …/twc/ ? ")
(case pick
(?e (my/to-emacs-config))
(?a (my/to-autrui))
(?g (my/to-gesta))
(?h (my/to-today))
(?r (my/to-readings))
(?w (my/to-writings))
(?t (my/to-twc))))

(global-set-key (kbd "C-x j ?") ‘my/pick-destination)

interactive "c… takes a single character (see further options here), so the conditional is on a character, which in Emacs Lisp is represented by a ? followed by the character {n}. If there were a single conditional we might use (when (char-equal ?e pick) {n}, but since there are seven of them, we look instead for Emacs Lisp’s equivalent of a switch statement, and find it in case (an alias for cl-case).

So what this does, when you type C-x j ?, is provide a prompt of options in the echo area at the bottom of the screen, and if you type one of the significant letters, it uses that shortcut to default.

Purists would probably prefer to bind ’my/pick-destination to something other than C-x j ? (C-x j j say), so that if we ever bound other commands to something starting with C-x j we would still be able to discover them with C-x j ?. It’s also easier to type because it doesn’t need the shift key for the third element. Having demonstrated that we could over-ride C-x j ? if we wanted to, I’m probably going to side with the purists on this one.

And we’re done. Commit.

ETA: Easier Ways To Do It

That works, but since (see Sacha’s comment below) there are always easier ways to do it:

Simpler Code

find-file will either open a file or open dired if it is passed a directory instead of a file, and opening a file or directory will change the default directory. So we can replace all the extracted ’my/to-file and ’my/to-dir methods with simple find-file calls:

(defun my/to-emacs-config ()
(find-file "~/.emacs.d/sean.org"))

(defun my/to-autrui ()
(find-file "~/code/autrui/"))

… etc …

Using Bookmarks (No Custom Code, Even Simpler)

While ’my/pick-destination gets around the non-interactive nature of ? (minibuffer-completion-help, e.g. C-c C-g ? for list of completions to C-c C-g), using straight-up bookmarks lets you do that without custom code:

C-x r m {name} RET ;; bookmark-set ;; sets a {named} bookmark at current location
C-x r b {name} RET ;; bookmark-jump ;; jump to {named} bookmark
C-x r l ;; list-bookmarks ;; list all bookmarks

So having set bookmarks in the same seven places, either files or directories, we could list-bookmarks and get

% Bookmark File
autrui ~/code/autrui/
emacs ~/.emacs.d/sean.org
gesta ~/Dropbox/gesta/
now ~/Dropbox/gesta/2015.org
readings ~/Dropbox/gesta/readings.org
twc ~/Dropbox/gesta/twc/
writings ~/Dropbox/gesta/writings.org

And this list is interactive. Or we could call C-x r b (bookmark-jump) and start typing the destination we want. (If we get bored of typing, we can reduce the bookmark names to single characters, remembering that C-x r l (list-bookmarks) will give us the translation table if we forget, and then we’re back to being five keystrokes away from any bookmark, without having to add any extra code.)

Using Bookmark+ (And Describing and Customizing Faces)

If we want to be able to do more advanced things with bookmarks – tag them, annotate them, rename them, run dired-like commands on the bookmark list – we can grab the Bookmark+ package and (require ’bookmark+) in our .emacs. (Having done that, if we press e by a line in the bookmark list, for instance, we can edit the lisp record for the bookmark, to rename it or change the destination or see how many times it has been used.)

One problem I had with bookmark+ is that the bookmark list was displaying illegibly in dark blue on a black background. To fix this, I needed to move the cursor over one of the dark blue on black bookmark names and type M-x describe-face, and it reported the face (Emacs-speak for style {n}) of the character, in this case Describe face (default `bmkp-local-file-without-region'):. Pressing enter took me to a buffer which described the face and provided a customize this face link at the end of the first line. I followed (pressed enter) that link to get to a customize face buffer which let me change the styling for that element. On the line:

[X] Foreground: blue [ Choose ] (sample)

I followed (pressed enter on) Choose, it opened a buffer of colour options, I scrolled up to a more visible one, pressed enter again, and got back to the customize face buffer, then went up to “Apply and Save” link and pressed enter again there. Going back to the bookmark list, the bookmarks were visible. The change is preserved for future sessions in the init.el file:

;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won’t work right.
‘(bmkp-local-file-without-region ((t (:foreground "green")))))

ETA 2: Using Hydra Instead

Heikki Lehvaslaiho suggested using hydra instead, and as the bookmark solution required a couple of extra keystrokes and I’d been curious about hydra anyway, I thought I’d give it a go.

(require ‘hydra)
(kbd "C-c j")
(defhydra hydra-jump (:color blue)
("e" (find-file "~/.emacs.d/sean.org") ".emacs.d")
("c" (find-file "~/.emacs.d/Cask") "Cask")

("a" (find-file "~/code/autrui/") "autrui")
("h" (find-file "~/Dropbox/gesta/2015.org") "hodie")
("r" (find-file "~/Dropbox/gesta/readings.org") "readings")
("w" (find-file "~/Dropbox/gesta/writings.org") "writings")
("t" (find-file "~/Dropbox/gesta/twc/") "twc")))

I like it. We’re back down to three keystrokes from five and, since we’re providing labels for the commands, we also get the descriptive and interactive and self-updating index which was the big advantage of the bookmark route. If you press C-c j and don’t immediately carry on, it prompts:

jump: [e]: .emacs.d, : Cask, [a]: autrui, [h]: hodie, [r]: readings, [w]: writings, [t]: twc.

Much better. Thanks for the prompt, Heikki!

Literate Emacs Configuration


Literate programming, proposed by Donald Knuth in 1984 {n}, suggests that

Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Literate Emacs configuration? Lets us build our .emacs.d files in .org files instead of .el files, so that with very little extra effort they can look like this.

In case that doesn’t speak for itself:

  • with org-mode’s headers and subheaders, you can quickly find your way around a large file, hiding all but the subheader content you care about
  • with org-mode’s links, the links to the snippets from What the .emacs.d!? or Avdi’s Emacs Reboot are live links
  • with org-mode’s export to html, you can weave an even more readable version of the config.
  • since it’s org-mode, you can put TODO statuses or tags on headers as well, so you can easily record the status of your practice on some shortcuts, or tag parts of the config as :experimental:.


The simplest version really does work out of the box.

In your init.el file, include an org-babel-load-file reference to the org file you’re going to use, in my case ~/.emacs.d/sean.org:

(org-babel-load-file "~/.emacs.d/sean.org")

Then, create the org file, and put whatever bits of config that you want in emacs-lisp source blocks, like so:

#+BEGIN_SRC emacs-lisp
  ... code here ...

What this will do is export the source code using org-babel-tangle (more on that later) into a file ~/.emacs.d/sean.el, and it will then load the resulting file using load-file. It is effectively the same as

(load-file "~/.emacs.d/sean.el")

except that it checks the timestamps of the sean.el and sean.org files and if the sean.org file was changed later, it re-extracts the sean.el file.

(Since sean.el is a generated file you will probably want to add it to your project’s .gitignore so it isn’t committed.)

And in the sean.org file you can put headings and subheadings to make it easy to navigate: in my first try, because my Emacs config had previously been across several files, I used the file names as subheadings, ending up with something like this:

* Emacs Config
** Defaults…
** Mode hooks…
** Global key bindings…
** Project-specific shortcuts…
** Zenburn theme…
** Emacs server and Emacsclient…

Check the Html Output

It is the case that if you export that from org to html right now (C-c C-e h o / org-export-dispatch) the h1 title will be the filename, which is unhelpfully just “sean”. Give it a better name with an explicit title export setting at the top of the file:

#+TITLE: Sean Miller’s Emacs Configuration

Re-export, and the html file has that title, and, by default, a linked table of contents derived from the headers and subheaders. (Other export options are described here.)

Run the Tests: A Catch

The next thing that I did was try to run the tests (tests introduced here, test runner introduced here), after changing the initial load-file line in the test to point to the generated sean.el file:

(load-file "sean.el")

And… I got an abnormal exit from ert-runner because the server-start from another part of my Emacs config prevented ert-runner from starting a server or running any of the tests.

We can fix that by breaking apart the emacs-lisp blocks from sean.org into two separate files, one of settings and one of code-under-test, and then in the test we can load the code-under-test file on its own.

To do this, we add a :tangle argument to the #+BEGIN_SRC emacs-lisp blocks in the sean.org file, with a filename argument. Code that is not under test gets the begin line:

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-settings.el

And code under test gets the begin line

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-code.el

Then, when we make changes to the sean.org file, on save we can run the org-babel-tangle command and it will export the emacs-lisp blocks into ~/.emacs.d/tangled-settings.el and ~/.emacs.d/tangled-code.el respectively.

We can then remove the org-babel-load-file line from init.el and replace it with:

(load-file "~/.emacs.d/tangled-settings.el")
(load-file "~/.emacs.d/tangled-code.el")

So that the next time we restart Emacs it will pick up the two built files, and if we change the load in our test file to be

(load-file "tangled-code.el")

And run our tests again, this time they run fine, so we have literate Emacs config and running tests.

(You’ll want to change your project’s .gitignore again to include the new “tangled-” files.)

Why “tangled”? It comes from the terminology of literate programming: the literate source file can be “tangled” to produce machine-readable code (exporting the source code blocks into .el files), and “woven” to produce formatted documentation. Knuth later admitted {n} that the echo of Walter Scott’s “Oh, what a tangled web we weave when first we practise to deceive” was entirely deliberate.

Automating the Manual Step We Just Introduced

When we fixed the catch we lost the automatic updating of the .el files when we changed the .org file. Now we need to remember to run org-babel-tangle manually after changing the .org file.

Manual steps are invariably forgotten, so let’s automate. The simplest hook would be:

(add-hook ‘after-save-hook ‘org-babel-tangle)

But we only want to tangle if we’ve just saved the Emacs config org file, so let’s build a new function to check that before running tangle, and hook to that instead:

(defun my/tangle-on-save-emacs-config-org-file()
(when (string= buffer-file-name (file-truename "~/.emacs.d/sean.org"))

(add-hook ‘after-save-hook ‘my/tangle-on-save-emacs-config-org-file)

buffer-file-name returns the absolute path of the name of the file the buffer is visiting. file-truename converts ~/.emacs.d/sean.org into an absolute path so that the string comparison works.

And with that, we’re done. Commit.

Trailing Whitespace in Commits

The Problem

You’re opening a pull request to do a code review. There are 34 changed files. You find this distressing, but you dig in and find only three changed files contain changes in logic: the rest are removing whitespace. You examine the three easily and go on to what’s next.

But suppose you had seen there were 34 changed files and cringed and gone on to another task instead? It would be better if there were a way to quickly distinguish real changes from formatting changes. Better still if we could prevent commits that add whitespace we’re only going to take out later in the first place.

Hiding but Not Solving the Problem

git diff, of course, shows you the differences between two code trees.

git diff -w / git diff --ignore-all-space shows you the differences excluding whitespace differences.

You can get github to ignore whitespace in its diffs by appending ?w=1 to the URL (say, of the pull request that you’re reviewing).

This solves the cognitive load problem – being able to tell at a glance that 31 of the 34 changed files are not significant is a huge plus – but all of those extra files have still been changed. Given a small team of great developers, no problem, but if you’ve got multiple people at multiple skill levels working on multiple branches, merges become more complicated and merge conflicts more likely than they need to be.

Solving the Problem: Going Forward

Any editor any developer on your team is likely to be using (Emacs, Vim, RubyMine, Sublime Text, TextMate) already has the ability to automatically remove trailing whitespace before saving a file. Make sure they all have it switched on. For Emacs, just add this to your config:

(add-hook ‘before-save-hook ‘delete-trailing-whitespace)

For other editors, there are instructions here.

Solving the Problem: Everything before That

If the project already has three years of commits before you switched that on, there will still be a problem with already-committed trailing whitespace.

What I would do in that case is find the least disruptive moment, grab something like rstrip (which is more flexible than one-liners from the command line because it starts by setting up a config file so you can decide which file extensions are processed), run it on the whole project, and commit the result. This will make you very unpopular with people on outlying branches, hence the importance of finding the least disruptive moment, but after that, you’ve fixed the trailing whitespace problem for good.

Teaser for Another Problem

Trailing whitespace is a single instance of the larger problem of automatically enforcing coding style conventions.

Back in Java this was a solved problem: write code with Eclipse, use the Checkstyle and Jalopy plugins (or their IntelliJ equivalents): Checkstyle creates warnings whenever a coding style convention is breached, and can also stop you committing code before resolving those warnings. Invoking Jalopy on Checkstyled code can automatically reformat it to follow the Checkstyle conventions to make it committable.

In Ruby? Thoughtbot has released hound which automatically comments on pull requests with Ruby / Javascript / Coffeescript / SCSS style coding style violations, based on Thoughtbot’s style guides.

Bozhidar Batsov (bbatsov) has a Ruby static code analyzer called rubocop which can be run locally and will produce warnings based on his community-driven Ruby style guide.

Yorick Peterse has another Ruby static code analyzer called ruby-lint.

Investigating all these is a large enough topic for another post.

Emacs Org-mode: More on Todos and Clock Reports

Previously, on The Wandering Coder, we used org-mode to build a more automatable answer to the questions “what did I work on last Friday?” and “what sorts of tasks did I work on last week?”. At the end of that entry, we had a simple file that let us clock time against tasks, set the TODO status of those tasks, and produce clock table reports of the result.

Here are three additional points that were not on that critical path but are still nice-to-haves:

Customizing TODO Keywords

TODO and DONE are the default TODO keywords, but you can over-ride that by creating a line #+TODO at the top of the file with your own list of custom todo keywords, for instance:


And then, after pressing C-c C-c to reload and use the new values, C-c C-t will run through not the default of TODO and DONE but the new list of custom keywords.

Annotating TODO State Transitions

If you have a custom #+TODO line, you can add a (!) or an (@) to annotate state transitions, for instance,


The (!) indicates that when a task is toggled into STARTED, the code automatically records a timestamp at the time of the state change.

The (@) indicates that not only does it record a timestamp when a task is toggled into DONE, it prompts the user for a note about the state change.

So if we had had that todo list in place for

** back-end task #121

and toggled it through TODO to STARTED to DONE, and added a note for the DONE state, we would end up with:

** DONE back-end task #121
– State "DONE" from "STARTED" [2015-02-06 Fri 15:00] \\
Note about task being done.
– State "STARTED" from "TODO" [2015-02-06 Fri 9:00]
CLOCK: [2015-02-06 Fri 9:00]–[2015-02-06 Fri 12:00] => 3:00
CLOCK: [2015-02-06 Fri 13:00]–[2015-02-06 Fri 15:00] => 2:00

More org-clock-table customizations

Exploring the clock table documentation reveals many more options to add to the #+BEGIN: line to customize the results than just :block today to see what was done today, or :block today-1 to see yesterday, or :tags “unplanned” to see the tasks tagged as :unplanned:.

If for instance we want the report for last week but broken down day-by-day, for instance, we could add to the #+BEGIN: line (remember also to press C-c C-c to update the results):

:block lastweek :step day

and get something like:

Daily report: [2015-02-09 Mon]
| Headline | Time | |
| *Total time* | *7:00* | |
| Work | 7:00 | |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |

Daily report: [2015-02-10 Tue]
| Headline | Time | |
| *Total time* | *7:00* | |
| Work | 7:00 | |
| \emsp back-end task #125 | | 7:00 |

Daily report: [2015-02-11 Wed]
| Headline | Time | |
| *Total time* | *5:00* | |
| Work | 5:00 | |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |


If your work sprint runs from Wednesday to Wednesday instead of Monday to Friday, as mine did last year, you can add :wstart 3 (instead of 1, which is Monday and the default), and then the weekly report from

:wstart 3 :block lastweek :step day

would begin instead with

Daily report: [2015-02-11 Wed]

Emacs Documentation: Info-mode, Describe-mode

Links to Emacs Lisp or Org-mode documentation in the blog are to the online versions at https://www.gnu.org/software/emacs/manual/html_node/elisp/ and http://orgmode.org/manual/, but you can also read the manuals within Emacs.

C-h i opens an info-mode buffer with a list of available manuals, and if you scroll down to Elisp and press enter you can read the Emacs Lisp Reference Manual, or to Org Mode and press enter you can read the Org Mode Manual.

To determine what key shortcuts are available in a buffer, say because you’re in an info-mode buffer deep in the Org Mode Manual and you’re trying to get back up to the list of manuals, press C-h m to open another buffer in describe-mode, which will explain that from info-mode a ^ goes up a from a node to its parent node, and, when you’re finished, q quits and restores the info-mode buffer to whatever was in it before you started reading the manuals.

C-h m results are customized to the type of buffer and the modes active within it: from the org-mode buffer I’m typing this in, for instance, they include:

C-c M-p d org2blog/wp-post-buffer
C-c M-p p org2blog/wp-post-buffer-and-publish

M-s M-e my/begin-end-src-emacs-lisp
M-s M-q my/begin-end-quote