Literate Emacs Configuration

What?

Literate programming, proposed by Donald Knuth in 1984 {n}, suggests that

Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Literate Emacs configuration? Lets us build our .emacs.d files in .org files instead of .el files, so that with very little extra effort they can look like this.

In case that doesn’t speak for itself:

with org-mode’s headers and subheaders, you can quickly find your way around a large file, hiding all but the subheader content you care about
with org-mode’s links, the links to the snippets from What the .emacs.d!? or Avdi’s Emacs Reboot are live links
with org-mode’s export to html, you can weave an even more readable version of the config.
since it’s org-mode, you can put TODO statuses or tags on headers as well, so you can easily record the status of your practice on some shortcuts, or tag parts of the config as :experimental:.

How?

The simplest version really does work out of the box.

In your init.el file, include an org-babel-load-file reference to the org file you’re going to use, in my case ~/.emacs.d/sean.org:

(org-babel-load-file "~/.emacs.d/sean.org")

Then, create the org file, and put whatever bits of config that you want in emacs-lisp source blocks, like so:

#+BEGIN_SRC emacs-lisp
  ... code here ...
#+END_SRC

What this will do is export the source code using org-babel-tangle (more on that later) into a file ~/.emacs.d/sean.el, and it will then load the resulting file using load-file. It is effectively the same as

(load-file "~/.emacs.d/sean.el")

except that it checks the timestamps of the sean.el and sean.org files and if the sean.org file was changed later, it re-extracts the sean.el file.

(Since sean.el is a generated file you will probably want to add it to your project’s .gitignore so it isn’t committed.)

And in the sean.org file you can put headings and subheadings to make it easy to navigate: in my first try, because my Emacs config had previously been across several files, I used the file names as subheadings, ending up with something like this:

* Emacs Config
** Defaults…
** Mode hooks…
** Global key bindings…
** Project-specific shortcuts…
** Zenburn theme…
** Emacs server and Emacsclient…
…etc

Check the Html Output

It is the case that if you export that from org to html right now (C-c C-e h o / org-export-dispatch) the h1 title will be the filename, which is unhelpfully just “sean”. Give it a better name with an explicit title export setting at the top of the file:

#+TITLE: Sean Miller’s Emacs Configuration

Re-export, and the html file has that title, and, by default, a linked table of contents derived from the headers and subheaders. (Other export options are described here.)

Run the Tests: A Catch

The next thing that I did was try to run the tests (tests introduced here, test runner introduced here), after changing the initial load-file line in the test to point to the generated sean.el file:

(load-file "sean.el")

And… I got an abnormal exit from ert-runner because the server-start from another part of my Emacs config prevented ert-runner from starting a server or running any of the tests.

We can fix that by breaking apart the emacs-lisp blocks from sean.org into two separate files, one of settings and one of code-under-test, and then in the test we can load the code-under-test file on its own.

To do this, we add a :tangle argument to the #+BEGIN_SRC emacs-lisp blocks in the sean.org file, with a filename argument. Code that is not under test gets the begin line:

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-settings.el

And code under test gets the begin line

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-code.el

Then, when we make changes to the sean.org file, on save we can run the org-babel-tangle command and it will export the emacs-lisp blocks into ~/.emacs.d/tangled-settings.el and ~/.emacs.d/tangled-code.el respectively.

We can then remove the org-babel-load-file line from init.el and replace it with:

(load-file "~/.emacs.d/tangled-settings.el")
(load-file "~/.emacs.d/tangled-code.el")

So that the next time we restart Emacs it will pick up the two built files, and if we change the load in our test file to be

(load-file "tangled-code.el")

And run our tests again, this time they run fine, so we have literate Emacs config and running tests.

(You’ll want to change your project’s .gitignore again to include the new “tangled-” files.)

Why “tangled”? It comes from the terminology of literate programming: the literate source file can be “tangled” to produce machine-readable code (exporting the source code blocks into .el files), and “woven” to produce formatted documentation. Knuth later admitted {n} that the echo of Walter Scott’s “Oh, what a tangled web we weave when first we practise to deceive” was entirely deliberate.

Automating the Manual Step We Just Introduced

When we fixed the catch we lost the automatic updating of the .el files when we changed the .org file. Now we need to remember to run org-babel-tangle manually after changing the .org file.

Manual steps are invariably forgotten, so let’s automate. The simplest hook would be:

(add-hook ‘after-save-hook ‘org-babel-tangle)

But we only want to tangle if we’ve just saved the Emacs config org file, so let’s build a new function to check that before running tangle, and hook to that instead:

(defun my/tangle-on-save-emacs-config-org-file()
(when (string= buffer-file-name (file-truename "~/.emacs.d/sean.org"))
(org-babel-tangle)))

(add-hook ‘after-save-hook ‘my/tangle-on-save-emacs-config-org-file)

buffer-file-name returns the absolute path of the name of the file the buffer is visiting. file-truename converts ~/.emacs.d/sean.org into an absolute path so that the string comparison works.

And with that, we’re done. Commit.

Trailing Whitespace in Commits

The Problem

You’re opening a pull request to do a code review. There are 34 changed files. You find this distressing, but you dig in and find only three changed files contain changes in logic: the rest are removing whitespace. You examine the three easily and go on to what’s next.

But suppose you had seen there were 34 changed files and cringed and gone on to another task instead? It would be better if there were a way to quickly distinguish real changes from formatting changes. Better still if we could prevent commits that add whitespace we’re only going to take out later in the first place.

Hiding but Not Solving the Problem

git diff, of course, shows you the differences between two code trees.

git diff -w / git diff --ignore-all-space shows you the differences excluding whitespace differences.

You can get github to ignore whitespace in its diffs by appending ?w=1 to the URL (say, of the pull request that you’re reviewing).

This solves the cognitive load problem – being able to tell at a glance that 31 of the 34 changed files are not significant is a huge plus – but all of those extra files have still been changed. Given a small team of great developers, no problem, but if you’ve got multiple people at multiple skill levels working on multiple branches, merges become more complicated and merge conflicts more likely than they need to be.

Solving the Problem: Going Forward

Any editor any developer on your team is likely to be using (Emacs, Vim, RubyMine, Sublime Text, TextMate) already has the ability to automatically remove trailing whitespace before saving a file. Make sure they all have it switched on. For Emacs, just add this to your config:

(add-hook ‘before-save-hook ‘delete-trailing-whitespace)

For other editors, there are instructions here.

Solving the Problem: Everything before That

If the project already has three years of commits before you switched that on, there will still be a problem with already-committed trailing whitespace.

What I would do in that case is find the least disruptive moment, grab something like rstrip (which is more flexible than one-liners from the command line because it starts by setting up a config file so you can decide which file extensions are processed), run it on the whole project, and commit the result. This will make you very unpopular with people on outlying branches, hence the importance of finding the least disruptive moment, but after that, you’ve fixed the trailing whitespace problem for good.

Teaser for Another Problem

Trailing whitespace is a single instance of the larger problem of automatically enforcing coding style conventions.

Back in Java this was a solved problem: write code with Eclipse, use the Checkstyle and Jalopy plugins (or their IntelliJ equivalents): Checkstyle creates warnings whenever a coding style convention is breached, and can also stop you committing code before resolving those warnings. Invoking Jalopy on Checkstyled code can automatically reformat it to follow the Checkstyle conventions to make it committable.

In Ruby? Thoughtbot has released hound which automatically comments on pull requests with Ruby / Javascript / Coffeescript / SCSS style coding style violations, based on Thoughtbot’s style guides.

Bozhidar Batsov (bbatsov) has a Ruby static code analyzer called rubocop which can be run locally and will produce warnings based on his community-driven Ruby style guide.

Yorick Peterse has another Ruby static code analyzer called ruby-lint.

Investigating all these is a large enough topic for another post.

Emacs Org-mode: More on Todos and Clock Reports

Previously, on The Wandering Coder, we used org-mode to build a more automatable answer to the questions “what did I work on last Friday?” and “what sorts of tasks did I work on last week?”. At the end of that entry, we had a simple file that let us clock time against tasks, set the TODO status of those tasks, and produce clock table reports of the result.

Here are three additional points that were not on that critical path but are still nice-to-haves:

Customizing TODO Keywords

TODO and DONE are the default TODO keywords, but you can over-ride that by creating a line #+TODO at the top of the file with your own list of custom todo keywords, for instance:

#+TODO: TODO STARTED BLOCKED QA DONE

And then, after pressing C-c C-c to reload and use the new values, C-c C-t will run through not the default of TODO and DONE but the new list of custom keywords.

Annotating TODO State Transitions

If you have a custom #+TODO line, you can add a (!) or an (@) to annotate state transitions, for instance,

#+TODO: TODO STARTED(!) DONE(@)

The (!) indicates that when a task is toggled into STARTED, the code automatically records a timestamp at the time of the state change.

The (@) indicates that not only does it record a timestamp when a task is toggled into DONE, it prompts the user for a note about the state change.

So if we had had that todo list in place for

** back-end task #121

and toggled it through TODO to STARTED to DONE, and added a note for the DONE state, we would end up with:

** DONE back-end task #121
– State "DONE" from "STARTED" [2015-02-06 Fri 15:00] \\
Note about task being done.
– State "STARTED" from "TODO" [2015-02-06 Fri 9:00]
CLOCK: [2015-02-06 Fri 9:00]–[2015-02-06 Fri 12:00] => 3:00
CLOCK: [2015-02-06 Fri 13:00]–[2015-02-06 Fri 15:00] => 2:00

More org-clock-table customizations

Exploring the clock table documentation reveals many more options to add to the #+BEGIN: line to customize the results than just :block today to see what was done today, or :block today-1 to see yesterday, or :tags “unplanned” to see the tasks tagged as :unplanned:.

If for instance we want the report for last week but broken down day-by-day, for instance, we could add to the #+BEGIN: line (remember also to press C-c C-c to update the results):

:block lastweek :step day

and get something like:

Daily report: [2015-02-09 Mon]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |

Daily report: [2015-02-10 Tue]
| Headline | Time | |
|————————–+——–+——|
| *Total time* | *7:00* | |
|————————–+——–+——|
| Work | 7:00 | |
| \emsp back-end task #125 | | 7:00 |

Daily report: [2015-02-11 Wed]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *5:00* | |
|—————————+——–+——|
| Work | 5:00 | |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |

etc…

If your work sprint runs from Wednesday to Wednesday instead of Monday to Friday, as mine did last year, you can add :wstart 3 (instead of 1, which is Monday and the default), and then the weekly report from

:wstart 3 :block lastweek :step day

would begin instead with

Daily report: [2015-02-11 Wed]

Emacs Documentation: Info-mode, Describe-mode

Links to Emacs Lisp or Org-mode documentation in the blog are to the online versions at https://www.gnu.org/software/emacs/manual/html_node/elisp/ and http://orgmode.org/manual/, but you can also read the manuals within Emacs.

C-h i opens an info-mode buffer with a list of available manuals, and if you scroll down to Elisp and press enter you can read the Emacs Lisp Reference Manual, or to Org Mode and press enter you can read the Org Mode Manual.

To determine what key shortcuts are available in a buffer, say because you’re in an info-mode buffer deep in the Org Mode Manual and you’re trying to get back up to the list of manuals, press C-h m to open another buffer in describe-mode, which will explain that from info-mode a ^ goes up a from a node to its parent node, and, when you’re finished, q quits and restores the info-mode buffer to whatever was in it before you started reading the manuals.

C-h m results are customized to the type of buffer and the modes active within it: from the org-mode buffer I’m typing this in, for instance, they include:

C-c M-p d org2blog/wp-post-buffer
C-c M-p p org2blog/wp-post-buffer-and-publish
etc…

M-s M-e my/begin-end-src-emacs-lisp
M-s M-q my/begin-end-quote
etc…

Emacs Org-mode: Tasks, Todos and Timings

The Problem

You’re in a Monday morning standup meeting. Everybody’s update begins “Well, I don’t really remember what I did on Friday, but…”

The Simplest Solution

In a notebook or a text file, write it down. Add a line for each task or ticket you worked on in a day.

– Friday 6 February
– back-end task #122
– production bug #123
– front-end task #124

This solves the Monday-morning-I-don’t-remember-Friday problem, but what if you’re trying to get better at estimating and you want to know how long each task took? What if you want a report on how much of it was planned tasks versus unplanned interruptions? And how do you automate as much of this as possible?

Enter Org-mode

From the first line of the manual:

Org is a mode for keeping notes, maintaining TODO lists, and doing project planning with a fast and effective plain-text system.

Create a file called work.org. (The .org suffix will automatically put it in org-mode when reading it in Emacs.)

* Work
** back-end task #121
** back-end task #122
** production bug #123
** front-end task #124

Outline

The file is an outline: lines starting with asterisks are headers, and the more asterisks, the deeper the sub-header, so in html terms (or explicitly if you export an org-mode file to html):

* h1 header ;; => <h1>h1 header<//h1>
** h2 sub-header ;; => <h2>h2 sub-header</h2>

Any line not starting with an asterisk is content contained within the header above it, so you can start adding notes about production bug #123:

* Work
** back-end task #121
** back-end task #122
** production bug #123

Affecting servers X, Y
Tried A, B, C

** front-end-task #124

To hide anything under a header (notes or any subheaders) you can press tab at the beginning of the line, so to hide the details under “** production bug #123” you would press tab at the beginning of the “** production bug #123” line, which would change the display to:

* Work
** back-end task #121
** back-end task #122
** production bug #123…
** front-end-task #124

Pressing tab again at the beginning of the “** production bug #123…” line will re-reveal the hidden content.

You can also press shift-tab anywhere to cycle between displaying just the top-level header, all the headers and subheaders, and all the headers, subheaders, and text.

Tracking time against tasks

You arrive Friday morning, look at your list, and start work on back-end task #121. To start tracking the time spent, go to the header back-end tasks #121 and press C-c C-x C-i, which runs org-clock-in, and starts a clock on that header.

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]

When you finish the task or stop working on it, press C-c C-x C-o to run org-clock-out, which will stop the running clock on this task:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:00] => 2:00

Alternately, if you run org-clock-in on another task, it will stop the clock on the previous task before starting it on the new one:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 10:45] => 1:45
** back-end task #122
CLOCK: [2015-02-06 Fri 10:45]

If you leave a task and come back to it (after lunch, say), you can clock out when you leave and clock back in again when you come back, and it starts a new clock:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 12:45]
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:45] => 2:45

When multiple clock records become distracting, remember that pressing tab at the beginning of a header hides everything underneath that header, so the last example can be reduced to:

* Work
** back-end task #121…

Reporting

You’ve built up your list for a week, adding new tasks as you go and clocking in and out of them to keep track of time spent. You come in on Monday the 16th and it’s time for morning standup. To get from the list of tasks a report of what you worked on on Friday the 13th, you have two options: use org-clock-report or org-agenda. We’ll discuss org-agenda in a later post.

C-c C-x C-r on a header runs org-clock-report on all the content below that header, so running it here on the * Work header would autogenerate a report of all the times for all the tasks, which over a week might look like:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55]
| Headline | Time | |
|—————————+————+——|
| *Total time* | *1d 16:00* | |
|—————————+————+——|
| Work | 1d 16:00 | |
| \emsp back-end task #121 | | 5:00 |
| \emsp back-end task #122 | | 2:00 |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |
| \emsp back-end task #125 | | 7:00 |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |
| \emsp hotfix #128 | | 1:00 |
| \emsp front-end task #129 | | 8:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

In this case, that’s all the tasks for eight days, not all the tasks just for last Friday. To limit it, we can modify the auto-generated #+BEGIN line by adding :block today-3 at the end. Press C-c C-c on that line to update the table, and we get instead:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree :block today-3
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp front-end task #129 | | 2:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

And there’s our standup report. Well, almost. It would be nice if it also told us whether each task was done or not.

Status of tasks

As part of org-mode’s handling of TODO lists, any header can be turned into a TODO item. With the cursor on a header line, press C-c C-t (org-todo) three times. It will cycle from (say)

** back-end task #121

** TODO back-end task #121

** DONE back-end task #121

and back to

** back-end task #121

And the TODO and DONE fields show up in the clock reports, so if these TODO keywords were set and front-end task #129 were DONE and front-end task #130 were TODO, the report would look instead like

#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|——————————–+——–+——|
| *Total time* | *7:00* | |
|——————————–+——–+——|
| Work | 7:00 | |
| \emsp DONE front-end task #129 | | 2:00 |
| \emsp TODO front-end task #130 | | 5:00 |

And there’s our standup report.

Planned versus Unplanned: Tagging Tasks

The one other thing in our initial problem was to be able to report on planned tasks versus unplanned interruptions, perhaps as part of figuring out velocity or how much unexpected work or emergencies disrupted the work of the sprint. To handle this we can add tags to headers, and use those as filters for the org-clock-report.

Suppose

** production bug #123
** production bug #126
** hotfix #128

are unplanned interruptions. With the cursor on those headers, press C-c C-q to add a tag. It will prompt for a tag name: you can just type in unplanned in each case or set up a list of possible tags like so

#+TAGS: unplanned

and then (after reloading with C-c C-c), you get autocompletion on tag names. Having added the tags, the collapsed table will look like:

** back-end task #121…
** back-end task #122…
** production bug #123 :unplanned:…
** front-end task #124…
** back-end task #125…
** production bug #126 :unplanned:…
** back-end task #127…
** hotfix #128 :unplanned:…
** DONE front-end task #129…
** TODO front-end task #130…

At that point, you can use org-clock-mode using :tags on the #+BEGIN: line and the tag match syntax, so you can retrieve unplanned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|—————————+——–+——| |
| *Total time* | *5:00* | | |
|—————————+——–+——| |
| Work | 5:00 | | |
| \emsp production bug #123 | | 2:00 | |
| \emsp production bug #126 | | 2:00 | |
| \emsp hotfix #128 | | 1:00 | |
#+END:

and planned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "-unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|——————————–+————+——| |
| *Total time* | *1d 11:00* | | |
|——————————–+————+——| |
| Work | 1d 11:00 | | |
| \emsp back-end task #121 | | 5:00 | |
| \emsp back-end task #122 | | 2:00 | |
| \emsp front-end task #124 | | 5:00 | |
| \emsp back-end task #125 | | 7:00 | |
| \emsp back-end task #127 | | 3:00 | |
| \emsp DONE front-end task #129 | | 8:00 | |
| \emsp TODO front-end task #130 | | 5:00 | |
#+END: |

And there, in addition to our standup report, is report on planned tasks vs. unplanned interruptions. The entire initial problem is now solved.

Refactoring “Beginning Emacs Lisp”: II

Now that we have test coverage and a test runner, we can refactor the begin/end code.

First off, let’s namespace it. Instead of defun begin-end-quote (), let’s make it defun my/begin-end-quote ().

Why? There is another library with a function called begin-end-quote: I’m not using it, but if I were, after namespacing I wouldn’t have to worry about anything breaking because of the name collision.

The coding conventions suggest namespacing packages for widespread use with -, not / (q.v. typopunct-mode ’s typopunct-insert-quotation-mark), but namespacing “less formal” code with / seems a useful distinction. Sacha Chua suggested a further convention for Emacs config files of my/ instead of (say) sean/, since .emacs.d code is so often copied and borrowed and it would make it look more consistent after mixing code from multiple sources.

Run the tests: they still pass. Hurrah.

Almost all of the code between begin-end-verse and begin-end-quote is the same. I made them separate to begin with because I thought they would be more different, but they aren’t. So we can make everything but the outermost function take an argument of “quote” or “verse” and remove the duplication. (This will also let me quickly add variants like #+begin_src emacs-lisp and #+begin_example, which I’ve already needed while writing this blog.)

Again, run the tests, they still pass. Commit with this change.

Further quick points: newline does insert a carriage return, but insert “\n” would do the same thing, and feels more intuitive. Likewise, while previous-line and next-line do what they say, the more usual way would be to run forward-line with a positive or negative integer argument.

There is also duplication between my/begin-end-selected-region and my/begin-end-no-selected-region: they both print the #+begin and #+end tags, and if there’s a selected region it is reformatted, and if there isn’t, after printing the tags it moves the cursor back up between the tags. We can collapse the two methods into a single method with two conditionals.

(defun my/begin-end (variant)
(interactive)
(let ((cited-string "\n"))
(when (use-region-p)
(setq cited-string
(my/remove-old-citation-formatting (buffer-substring-no-properties (region-beginning) (region-end))))
(delete-region (region-beginning) (region-end)))
(insert "#+begin_" variant "\n"
cited-string
"#+end_" variant "\n"))
(unless (use-region-p)
(forward-line -2)))

The simpler conditional is at lines 11-12: (use-region-p) returns true if there is a region selected, and if there isn’t, it moves the cursor up two lines.

The one in lines 3-10 is more complicated.

In lines 3-7, we’re using a let statement to set cited-string initially to \n, which is what it should be if there’s no selection, and then we’re checking to see if there is a selected region: if there is, we’re resetting cited-string to the value of the selected region with the old formatting code stripped out, and then we’re deleting the selected region.

At the beginning of line 8, then, we’ve deleted the selected region if there was one, and until the end of the body of the let at the end of line 10, cited-string contains either a return character (if there was no selection) or the reformatted contents of the selected region (if there was a selection). Because of that, we can insert it between the tags either way.

As part of this, we also modified the my/fix-old-formatting method from one that dealt with the selected region in place to one which took a string and returned a modified string. All the selected region manipulation now takes place in the my/begin-end function.

Re-run the tests: they still pass. Excellent. Commit.

At this point, the refactoring is done, but it is now trivially easy to add new tests and implementations and key-bindings for #+begin_example / #+end_example and #+begin_src emacs-lisp / #+end_src.

The test for #+begin_src emacs-lisp breaks because for _src, the beginning tag and the end tag are different. The simplest way to get this passing is to pass in two arguments to my/begin-end, which for all but src will be identical, like so:

(defun my/begin-end-example ()
(interactive)
(my/begin-end "example" "example"))

(defun my/begin-end-src-emacs-lisp ()
(interactive)
(my/begin-end "src emacs-lisp" "src"))

(defun my/begin-end (begin-tag end-tag)
…
(insert "#+begin_" begin-tag "\n"
cited-string
"#+end_" end-tag "\n"))

At this point, the code does everything we need, the tests pass, and we drove out the new functionality we needed with tests. We’re done. Final commit.

Emacs Lisp: Adding Tests: ert-runner and overseer

In the previous Adding Tests post I found that running M-x ert on its own did not always pick up added or removed tests, but that a test running tool called ert-runner might fix this. It does, and adding another tool called overseer makes it easy to run tests or subsets of tests without leaving Emacs.

Here’s how I got there:

Install Cask to manage project dependencies, as ert-runner uses it to run the tests from the command-line. I found that the default brew install cask didn’t set up ~/.cask/cask.el properly, but the longer curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python did.
- After that, the usage instructions mostly worked out of the tin.
  - Calling cask init --dev sets up a Cask file (think Gemfile from ruby projects) with a block of development dependencies. We aren’t using ecukes or el-mock yet, and we’ll add overseer.
  - I also took out the (package-file "TODO") line, as it stopped cask install from running.
  - I had borrowed package-loading from bbatsov’s excellent Prelude: for now I’ve commented that out and put my Emacs package dependencies into the Cask file.
  - Running cask install puts all the packages in a .cask directory, so you might want to add that to .gitignore.
  - commit for adding Cask
With ert-runner installed by Cask, from the command line, run cask exec ert-runner init. This creates a test directory and an empty test/test-helper.el file. Previously, our tests for the begin/end functionality were in the same file as the code: for ert-runner to be able to find them, they need to be in the test directory, in a file ending -test.el. So we move the tests from mods-org.el to test/mods-org-test.el.
- The newbie / coming from Ruby mistake that I made at this point was to go to the command line and run cask exec ert-runner, expecting it to pick up the files automatically. This of course fails. We need to put a (provide 'mods-org) at the bottom of the mods-org.el file, and a (load-file "mods-org.el") at the top of test/mods-org-test.el.
- Having done that, we can run the tests, and they all pass. Further, if some of them fail, we can do the usual thing: comment out all but one of them and re-run and, unlike with M-x ert, it picks up the changes and just runs a single test.
- commit for reorganizing tests
When going out to the command line or popping open a shell to run cask exec ert-runner every time proves irksome, we can take advantage of the fact that we also added overseer, which gives us some handy shortcuts. With a single test file so far, I have been using C-c , b to run all the tests from the buffer.

Thanks to Sacha Chua for unblocking me at one point, and sharing a draft post which takes things further into continuous integration and test coverage tools.

Emacs Org-mode: Publishing to WordPress

The next step on this emacs org-mode kick? Instead of editing the blog in markdown and publishing it with jekyll, edit in org-mode and publish to WordPress. There’s a package for this, org2blog/wp, and very useful setup instructions here.

The only change I had to make is that using .netrc as described returned an

Wrong type argument: stringp, netrc-get

error. Looking at the org2blog/wp package itself, I discovered that a commit five months after the instructions came out recommended replacing ‘netrc with ‘auth-source. After making that change, everything worked.

The commit to my .emacs.d including this change is here. The ~/.authinfo file is in the form:

machine thewanderingcoder
  login {my-login}
  password {my-password}

One further note: on trying both the native emacs highlighting (org2blog/wp-use-sourcecode-shortcode nil) and the SyntaxHighlighter Evolved plugin highlighting (org2blog/wp-use-sourcecode-shortcode ‘t), I had to agree that the SyntaxHighlighter Evolved plugin highlighting looked much better.

The one difficulty, particularly for a post about testing, is that it didn’t recognize and highlight ert-deftest. So I edited my copy of shBrushLisp.js and added ert-deftest to the end of the list of macros:

var macros = ‘loop when dolist dotimes defun ert-deftest’;

and uploaded the modified file to wp-content/plugins/syntaxhighlighter, and now ert-deftest is highlighted accordingly.

Refactoring “Beginning Emacs Lisp”: I: Adding Tests

On Friday I sat down with Sacha Chua for some emacs coaching. We talked about org-mode and about the emacs lisp I’d written for reformatting citations. In this entry I’ll talk about refactoring that emacs lisp code.

Ah, Refactoring. We’ll Need Some Tests…

Refactoring, by definition, is improving the internal structure of code without altering the external behaviour. Equally by definition, before you start you need thorough automated tests, because that’s how you tell that you haven’t altered the external behaviour.

I wrote the reformatting citations emacs lisp as an exploratory spike, looking up commands as I went, and manually testing the results. Time to get more rigorous. What is emacs lisp’s equivalent of JUnit, or MiniTest or RSpec?

ERT: Emacs Lisp Regression Testing

In JUnit you annotate test methods with “@Test”:

@Test
public void formatingRemoved {

In MiniTest, you start the method definitions with “test_”:

def test_formatting_removed

In ERT, where you would define a normal lisp function, with “defun”:

(defun remove-formatting ()
)

you can define a lisp test with “ert-deftest”:

(ert-deftest remove-formatting ()
)

Inside the test, you can call the function under test and compare the actual and expected results with the “should” macro. For instance, given a function that takes a string and removes some formatting (using replace-regexp-in-string):

(defun remove-formatting (string)
(replace-regexp-in-string "^> " ""
(replace-regexp-in-string "\s*<br/?>" "" string)))

We could write a test that checks that it does what it says:

(ert-deftest remove-formatting ()
(should (string= (remove-formatting "> Elþeodigra eard<br/>")
"Elþeodigra eard")))

To run all the tests, we can type:

M-x ert RET RET

The second RET accepts the default, t, and runs all tests. In this case there’s only one: if we have a large suite and we only want to run a subset, say those with “formatting” in the test name, we would type instead:

M-x ert RET "*formatting*" RET

In either case, another buffer is opened up with the results, for instance

Selector: t
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2015-02-02 16:03:15-0500
Finished.
Finished at:  2015-02-02 16:03:15-0500

.

And yes, as you would expect from other languages, that’s a dot per passing test, and an F for any failing test, so with twelve tests and two failures you might instead see:

..FF........

In the ERT results buffer, with the cursor on any . or F test result you have several options, including:

. ;; jump to that test’s source code
l ;; list the assertions run by the test
h ;; see the description string for the test, if any
b ;; view backtrace
r ;; re-run this test

As Nic Ferrier notes, the runner doesn’t automatically recognize when you delete a test, and you need to delete it in the ERT results buffer. I had trouble with it recognizing added tests as well, and ended up closing and restarting emacs to make sure it picked up the latest list. This is obviously not scalable. There’s a separate tool called ert-runner which fixes this, as I discover here.

Additional Complications

If we were testing code that took a string and reformatted it, like the example above, we could just write (should (string= examples for all the edge cases we could think of using what we already know, and we’d be set. Unfortunately, the code to be put under test also makes changes to the buffer, varies its behaviour depending on whether a region is selected when it is called, and modifies the cursor position. How do we handle that?

The with-temp-buffer macro saves the current buffer, creates an (empty) temporary buffer, marks it current, uses it inside the body of the macro, and on exit switches back to the previous current buffer. You can return the contents of the temporary buffer by using (buffer-string) as the last form, and the position of the cursor within the temporary buffer by using (point) as the last form. This lets us write tests for (begin-end-quote) when a region is not selected like so:

(ert-deftest test-begin-end-quote-new-content ()
"Tests begin-end-quote without preselected text string"
(should (string= (with-temp-buffer
(begin-end-quote)
(buffer-string))
"#+begin_quote\n\n#+end_quote\n")))

(ert-deftest test-begin-end-quote-new-point ()
"Tests begin-end-quote without preselected text cursor position"
(should (equal (with-temp-buffer
(begin-end-quote)
(point))
(length "#+begin_quote\n\n"))))

There’s a further complication for the case where a region is selected before calling it. We can include text in the temporary buffer before calling begin-end-quote by using insert, and then (set-mark .N.) to set the mark at the nth character, and then either goto-char .N. to the select the region from the first mark up to character n, or just do end-of-buffer to select the region to the end of the buffer. So to insert the text “> Dear Sir, your astonishment’s odd;\n” into the temporary buffer and select the whole region, we could do the following:

(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(end-of-buffer)

With that extra information, the tests of behaviour with a selected region become simple too:

(ert-deftest test-begin-end-quote-region ()
"Tests begin-end-quote with selected region"
(should (string= (with-temp-buffer
(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(buffer-string))
"#+begin_quote\n Dear Sir, your astonishment’s odd;\n#+end_quote\n")))

The commit with the full set of twelve tests is here.

As well as adding tests this commit makes a code change, because in the three weeks since writing it I have discovered that the archive files don’t always have exactly two spaces between the end of the text and the “<br/>”, so I wrote tests to expose that (so that two of them were failing, as in the example above), and then changed the regexp so that they passed.

Now that we’ve got full automated tests, we can start refactoring the code.

Emacs Org-mode: Links and Exported Html

If you have an archive of files in org-mode and you want to link between them, say from today’s entry to the entry of 9 March 2013, you have several options, as laid out here:

http://orgmode.org/manual/Search-options.html

The Simplest Case, for .org

The simplest is to provide the text of the header in the link, like so: so:

[[file:2013.org::Saturday 9 March 2013][9 March 2013]]

which, on typing the final closing square bracket, will collapse on screen to “9 March 2013”, and when you’re on the link and type “C-c C-o”, it will open the “2013.org” file at the header “Saturday 9 March 2013”. If you find you’ve made an error in the link target or title, typing “C-c C-l” will let you edit it.

http://orgmode.org/manual/Handling-links.html

And this works perfectly, until you export it to html, and then, of course, it doesn’t.

The Extra Step, for .html

To get a link also to work in html, you need to set a custom_id on the header, which you do like this:

*** Saturday 9 March 2013
:PROPERTIES:
:CUSTOM_ID: 20130309
:END:

or, less manually, with the org-set-property command (keyboard shortcut C-c C-x p):

C-c C-x p RET CUSTOM_ID RET 20130309

This will also attach an id attribute to the header element in the exported html, so in both .org and .html it is recognized as #20130309, and if you then change the link to

[[file:2013.org::#20130309][9 March 2013]]

it will work as before in the *.org file, and also in the exported *.html file.

The Final Step

Having done this a few times, you may get tired of typing “CUSTOM_ID” each time, and build a function that lets you just type the id value:

(defun cid (custom-id)
(interactive "MCUSTOM_ID: ")
(org-set-property "CUSTOM_ID" custom-id))

after which you can type, even more briefly:

M-x cid RET 20130309 RET

commit