Emacs Org-mode: More on Todos and Clock Reports

Previously, on The Wandering Coder, we used org-mode to build a more automatable answer to the questions “what did I work on last Friday?” and “what sorts of tasks did I work on last week?”. At the end of that entry, we had a simple file that let us clock time against tasks, set the TODO status of those tasks, and produce clock table reports of the result.

Here are three additional points that were not on that critical path but are still nice-to-haves:

Customizing TODO Keywords

TODO and DONE are the default TODO keywords, but you can over-ride that by creating a line #+TODO at the top of the file with your own list of custom todo keywords, for instance:

#+TODO: TODO STARTED BLOCKED QA DONE

And then, after pressing C-c C-c to reload and use the new values, C-c C-t will run through not the default of TODO and DONE but the new list of custom keywords.

Annotating TODO State Transitions

If you have a custom #+TODO line, you can add a (!) or an (@) to annotate state transitions, for instance,

#+TODO: TODO STARTED(!) DONE(@)

The (!) indicates that when a task is toggled into STARTED, the code automatically records a timestamp at the time of the state change.

The (@) indicates that not only does it record a timestamp when a task is toggled into DONE, it prompts the user for a note about the state change.

So if we had had that todo list in place for

** back-end task #121

and toggled it through TODO to STARTED to DONE, and added a note for the DONE state, we would end up with:

** DONE back-end task #121
– State "DONE" from "STARTED" [2015-02-06 Fri 15:00] \\
Note about task being done.
– State "STARTED" from "TODO" [2015-02-06 Fri 9:00]
CLOCK: [2015-02-06 Fri 9:00]–[2015-02-06 Fri 12:00] => 3:00
CLOCK: [2015-02-06 Fri 13:00]–[2015-02-06 Fri 15:00] => 2:00

More org-clock-table customizations

Exploring the clock table documentation reveals many more options to add to the #+BEGIN: line to customize the results than just :block today to see what was done today, or :block today-1 to see yesterday, or :tags “unplanned” to see the tasks tagged as :unplanned:.

If for instance we want the report for last week but broken down day-by-day, for instance, we could add to the #+BEGIN: line (remember also to press C-c C-c to update the results):

:block lastweek :step day

and get something like:

Daily report: [2015-02-09 Mon]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |

Daily report: [2015-02-10 Tue]
| Headline | Time | |
|————————–+——–+——|
| *Total time* | *7:00* | |
|————————–+——–+——|
| Work | 7:00 | |
| \emsp back-end task #125 | | 7:00 |

Daily report: [2015-02-11 Wed]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *5:00* | |
|—————————+——–+——|
| Work | 5:00 | |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |

etc…

If your work sprint runs from Wednesday to Wednesday instead of Monday to Friday, as mine did last year, you can add :wstart 3 (instead of 1, which is Monday and the default), and then the weekly report from

:wstart 3 :block lastweek :step day

would begin instead with

Daily report: [2015-02-11 Wed]

Emacs Documentation: Info-mode, Describe-mode

Links to Emacs Lisp or Org-mode documentation in the blog are to the online versions at https://www.gnu.org/software/emacs/manual/html_node/elisp/ and http://orgmode.org/manual/, but you can also read the manuals within Emacs.

C-h i opens an info-mode buffer with a list of available manuals, and if you scroll down to Elisp and press enter you can read the Emacs Lisp Reference Manual, or to Org Mode and press enter you can read the Org Mode Manual.

To determine what key shortcuts are available in a buffer, say because you’re in an info-mode buffer deep in the Org Mode Manual and you’re trying to get back up to the list of manuals, press C-h m to open another buffer in describe-mode, which will explain that from info-mode a ^ goes up a from a node to its parent node, and, when you’re finished, q quits and restores the info-mode buffer to whatever was in it before you started reading the manuals.

C-h m results are customized to the type of buffer and the modes active within it: from the org-mode buffer I’m typing this in, for instance, they include:

C-c M-p d org2blog/wp-post-buffer
C-c M-p p org2blog/wp-post-buffer-and-publish
etc…

M-s M-e my/begin-end-src-emacs-lisp
M-s M-q my/begin-end-quote
etc…

Emacs Org-mode: Tasks, Todos and Timings

The Problem

You’re in a Monday morning standup meeting. Everybody’s update begins “Well, I don’t really remember what I did on Friday, but…”

The Simplest Solution

In a notebook or a text file, write it down. Add a line for each task or ticket you worked on in a day.

– Friday 6 February
– back-end task #122
– production bug #123
– front-end task #124

This solves the Monday-morning-I-don’t-remember-Friday problem, but what if you’re trying to get better at estimating and you want to know how long each task took? What if you want a report on how much of it was planned tasks versus unplanned interruptions? And how do you automate as much of this as possible?

Enter Org-mode

From the first line of the manual:

Org is a mode for keeping notes, maintaining TODO lists, and doing project planning with a fast and effective plain-text system.

Create a file called work.org. (The .org suffix will automatically put it in org-mode when reading it in Emacs.)

* Work
** back-end task #121
** back-end task #122
** production bug #123
** front-end task #124

Outline

The file is an outline: lines starting with asterisks are headers, and the more asterisks, the deeper the sub-header, so in html terms (or explicitly if you export an org-mode file to html):

* h1 header ;; => <h1>h1 header<//h1>
** h2 sub-header ;; => <h2>h2 sub-header</h2>

Any line not starting with an asterisk is content contained within the header above it, so you can start adding notes about production bug #123:

* Work
** back-end task #121
** back-end task #122
** production bug #123

Affecting servers X, Y
Tried A, B, C

** front-end-task #124

To hide anything under a header (notes or any subheaders) you can press tab at the beginning of the line, so to hide the details under “** production bug #123” you would press tab at the beginning of the “** production bug #123” line, which would change the display to:

* Work
** back-end task #121
** back-end task #122
** production bug #123…
** front-end-task #124

Pressing tab again at the beginning of the “** production bug #123…” line will re-reveal the hidden content.

You can also press shift-tab anywhere to cycle between displaying just the top-level header, all the headers and subheaders, and all the headers, subheaders, and text.

Tracking time against tasks

You arrive Friday morning, look at your list, and start work on back-end task #121. To start tracking the time spent, go to the header back-end tasks #121 and press C-c C-x C-i, which runs org-clock-in, and starts a clock on that header.

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]

When you finish the task or stop working on it, press C-c C-x C-o to run org-clock-out, which will stop the running clock on this task:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:00] => 2:00

Alternately, if you run org-clock-in on another task, it will stop the clock on the previous task before starting it on the new one:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 10:45] => 1:45
** back-end task #122
CLOCK: [2015-02-06 Fri 10:45]

If you leave a task and come back to it (after lunch, say), you can clock out when you leave and clock back in again when you come back, and it starts a new clock:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 12:45]
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:45] => 2:45

When multiple clock records become distracting, remember that pressing tab at the beginning of a header hides everything underneath that header, so the last example can be reduced to:

* Work
** back-end task #121…

Reporting

You’ve built up your list for a week, adding new tasks as you go and clocking in and out of them to keep track of time spent. You come in on Monday the 16th and it’s time for morning standup. To get from the list of tasks a report of what you worked on on Friday the 13th, you have two options: use org-clock-report or org-agenda. We’ll discuss org-agenda in a later post.

C-c C-x C-r on a header runs org-clock-report on all the content below that header, so running it here on the * Work header would autogenerate a report of all the times for all the tasks, which over a week might look like:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55]
| Headline | Time | |
|—————————+————+——|
| *Total time* | *1d 16:00* | |
|—————————+————+——|
| Work | 1d 16:00 | |
| \emsp back-end task #121 | | 5:00 |
| \emsp back-end task #122 | | 2:00 |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |
| \emsp back-end task #125 | | 7:00 |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |
| \emsp hotfix #128 | | 1:00 |
| \emsp front-end task #129 | | 8:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

In this case, that’s all the tasks for eight days, not all the tasks just for last Friday. To limit it, we can modify the auto-generated #+BEGIN line by adding :block today-3 at the end. Press C-c C-c on that line to update the table, and we get instead:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree :block today-3
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp front-end task #129 | | 2:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

And there’s our standup report. Well, almost. It would be nice if it also told us whether each task was done or not.

Status of tasks

As part of org-mode’s handling of TODO lists, any header can be turned into a TODO item. With the cursor on a header line, press C-c C-t (org-todo) three times. It will cycle from (say)

** back-end task #121

** TODO back-end task #121

** DONE back-end task #121

and back to

** back-end task #121

And the TODO and DONE fields show up in the clock reports, so if these TODO keywords were set and front-end task #129 were DONE and front-end task #130 were TODO, the report would look instead like

#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|——————————–+——–+——|
| *Total time* | *7:00* | |
|——————————–+——–+——|
| Work | 7:00 | |
| \emsp DONE front-end task #129 | | 2:00 |
| \emsp TODO front-end task #130 | | 5:00 |

And there’s our standup report.

Planned versus Unplanned: Tagging Tasks

The one other thing in our initial problem was to be able to report on planned tasks versus unplanned interruptions, perhaps as part of figuring out velocity or how much unexpected work or emergencies disrupted the work of the sprint. To handle this we can add tags to headers, and use those as filters for the org-clock-report.

Suppose

** production bug #123
** production bug #126
** hotfix #128

are unplanned interruptions. With the cursor on those headers, press C-c C-q to add a tag. It will prompt for a tag name: you can just type in unplanned in each case or set up a list of possible tags like so

#+TAGS: unplanned

and then (after reloading with C-c C-c), you get autocompletion on tag names. Having added the tags, the collapsed table will look like:

** back-end task #121…
** back-end task #122…
** production bug #123 :unplanned:…
** front-end task #124…
** back-end task #125…
** production bug #126 :unplanned:…
** back-end task #127…
** hotfix #128 :unplanned:…
** DONE front-end task #129…
** TODO front-end task #130…

At that point, you can use org-clock-mode using :tags on the #+BEGIN: line and the tag match syntax, so you can retrieve unplanned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|—————————+——–+——| |
| *Total time* | *5:00* | | |
|—————————+——–+——| |
| Work | 5:00 | | |
| \emsp production bug #123 | | 2:00 | |
| \emsp production bug #126 | | 2:00 | |
| \emsp hotfix #128 | | 1:00 | |
#+END:

and planned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "-unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|——————————–+————+——| |
| *Total time* | *1d 11:00* | | |
|——————————–+————+——| |
| Work | 1d 11:00 | | |
| \emsp back-end task #121 | | 5:00 | |
| \emsp back-end task #122 | | 2:00 | |
| \emsp front-end task #124 | | 5:00 | |
| \emsp back-end task #125 | | 7:00 | |
| \emsp back-end task #127 | | 3:00 | |
| \emsp DONE front-end task #129 | | 8:00 | |
| \emsp TODO front-end task #130 | | 5:00 | |
#+END: |

And there, in addition to our standup report, is report on planned tasks vs. unplanned interruptions. The entire initial problem is now solved.

Refactoring “Beginning Emacs Lisp”: II

Now that we have test coverage and a test runner, we can refactor the begin/end code.

First off, let’s namespace it. Instead of defun begin-end-quote (), let’s make it defun my/begin-end-quote ().

Why? There is another library with a function called begin-end-quote: I’m not using it, but if I were, after namespacing I wouldn’t have to worry about anything breaking because of the name collision.

The coding conventions suggest namespacing packages for widespread use with -, not / (q.v. typopunct-mode ’s typopunct-insert-quotation-mark), but namespacing “less formal” code with / seems a useful distinction. Sacha Chua suggested a further convention for Emacs config files of my/ instead of (say) sean/, since .emacs.d code is so often copied and borrowed and it would make it look more consistent after mixing code from multiple sources.

Run the tests: they still pass. Hurrah.

Almost all of the code between begin-end-verse and begin-end-quote is the same. I made them separate to begin with because I thought they would be more different, but they aren’t. So we can make everything but the outermost function take an argument of “quote” or “verse” and remove the duplication. (This will also let me quickly add variants like #+begin_src emacs-lisp and #+begin_example, which I’ve already needed while writing this blog.)

Again, run the tests, they still pass. Commit with this change.

Further quick points: newline does insert a carriage return, but insert “\n” would do the same thing, and feels more intuitive. Likewise, while previous-line and next-line do what they say, the more usual way would be to run forward-line with a positive or negative integer argument.

There is also duplication between my/begin-end-selected-region and my/begin-end-no-selected-region: they both print the #+begin and #+end tags, and if there’s a selected region it is reformatted, and if there isn’t, after printing the tags it moves the cursor back up between the tags. We can collapse the two methods into a single method with two conditionals.

(defun my/begin-end (variant)
(interactive)
(let ((cited-string "\n"))
(when (use-region-p)
(setq cited-string
(my/remove-old-citation-formatting (buffer-substring-no-properties (region-beginning) (region-end))))
(delete-region (region-beginning) (region-end)))
(insert "#+begin_" variant "\n"
cited-string
"#+end_" variant "\n"))
(unless (use-region-p)
(forward-line -2)))

The simpler conditional is at lines 11-12: (use-region-p) returns true if there is a region selected, and if there isn’t, it moves the cursor up two lines.

The one in lines 3-10 is more complicated.

In lines 3-7, we’re using a let statement to set cited-string initially to \n, which is what it should be if there’s no selection, and then we’re checking to see if there is a selected region: if there is, we’re resetting cited-string to the value of the selected region with the old formatting code stripped out, and then we’re deleting the selected region.

At the beginning of line 8, then, we’ve deleted the selected region if there was one, and until the end of the body of the let at the end of line 10, cited-string contains either a return character (if there was no selection) or the reformatted contents of the selected region (if there was a selection). Because of that, we can insert it between the tags either way.

As part of this, we also modified the my/fix-old-formatting method from one that dealt with the selected region in place to one which took a string and returned a modified string. All the selected region manipulation now takes place in the my/begin-end function.

Re-run the tests: they still pass. Excellent. Commit.

At this point, the refactoring is done, but it is now trivially easy to add new tests and implementations and key-bindings for #+begin_example / #+end_example and #+begin_src emacs-lisp / #+end_src.

The test for #+begin_src emacs-lisp breaks because for _src, the beginning tag and the end tag are different. The simplest way to get this passing is to pass in two arguments to my/begin-end, which for all but src will be identical, like so:

(defun my/begin-end-example ()
(interactive)
(my/begin-end "example" "example"))

(defun my/begin-end-src-emacs-lisp ()
(interactive)
(my/begin-end "src emacs-lisp" "src"))

(defun my/begin-end (begin-tag end-tag)
…
(insert "#+begin_" begin-tag "\n"
cited-string
"#+end_" end-tag "\n"))

At this point, the code does everything we need, the tests pass, and we drove out the new functionality we needed with tests. We’re done. Final commit.

Emacs Lisp: Adding Tests: ert-runner and overseer

In the previous Adding Tests post I found that running M-x ert on its own did not always pick up added or removed tests, but that a test running tool called ert-runner might fix this. It does, and adding another tool called overseer makes it easy to run tests or subsets of tests without leaving Emacs.

Here’s how I got there:

Install Cask to manage project dependencies, as ert-runner uses it to run the tests from the command-line. I found that the default brew install cask didn’t set up ~/.cask/cask.el properly, but the longer curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python did.
- After that, the usage instructions mostly worked out of the tin.
  - Calling cask init --dev sets up a Cask file (think Gemfile from ruby projects) with a block of development dependencies. We aren’t using ecukes or el-mock yet, and we’ll add overseer.
  - I also took out the (package-file "TODO") line, as it stopped cask install from running.
  - I had borrowed package-loading from bbatsov’s excellent Prelude: for now I’ve commented that out and put my Emacs package dependencies into the Cask file.
  - Running cask install puts all the packages in a .cask directory, so you might want to add that to .gitignore.
  - commit for adding Cask
With ert-runner installed by Cask, from the command line, run cask exec ert-runner init. This creates a test directory and an empty test/test-helper.el file. Previously, our tests for the begin/end functionality were in the same file as the code: for ert-runner to be able to find them, they need to be in the test directory, in a file ending -test.el. So we move the tests from mods-org.el to test/mods-org-test.el.
- The newbie / coming from Ruby mistake that I made at this point was to go to the command line and run cask exec ert-runner, expecting it to pick up the files automatically. This of course fails. We need to put a (provide 'mods-org) at the bottom of the mods-org.el file, and a (load-file "mods-org.el") at the top of test/mods-org-test.el.
- Having done that, we can run the tests, and they all pass. Further, if some of them fail, we can do the usual thing: comment out all but one of them and re-run and, unlike with M-x ert, it picks up the changes and just runs a single test.
- commit for reorganizing tests
When going out to the command line or popping open a shell to run cask exec ert-runner every time proves irksome, we can take advantage of the fact that we also added overseer, which gives us some handy shortcuts. With a single test file so far, I have been using C-c , b to run all the tests from the buffer.

Thanks to Sacha Chua for unblocking me at one point, and sharing a draft post which takes things further into continuous integration and test coverage tools.

Emacs Org-mode: Publishing to WordPress

The next step on this emacs org-mode kick? Instead of editing the blog in markdown and publishing it with jekyll, edit in org-mode and publish to WordPress. There’s a package for this, org2blog/wp, and very useful setup instructions here.

The only change I had to make is that using .netrc as described returned an

Wrong type argument: stringp, netrc-get

error. Looking at the org2blog/wp package itself, I discovered that a commit five months after the instructions came out recommended replacing ‘netrc with ‘auth-source. After making that change, everything worked.

The commit to my .emacs.d including this change is here. The ~/.authinfo file is in the form:

machine thewanderingcoder
  login {my-login}
  password {my-password}

One further note: on trying both the native emacs highlighting (org2blog/wp-use-sourcecode-shortcode nil) and the SyntaxHighlighter Evolved plugin highlighting (org2blog/wp-use-sourcecode-shortcode ‘t), I had to agree that the SyntaxHighlighter Evolved plugin highlighting looked much better.

The one difficulty, particularly for a post about testing, is that it didn’t recognize and highlight ert-deftest. So I edited my copy of shBrushLisp.js and added ert-deftest to the end of the list of macros:

var macros = ‘loop when dolist dotimes defun ert-deftest’;

and uploaded the modified file to wp-content/plugins/syntaxhighlighter, and now ert-deftest is highlighted accordingly.

Refactoring “Beginning Emacs Lisp”: I: Adding Tests

On Friday I sat down with Sacha Chua for some emacs coaching. We talked about org-mode and about the emacs lisp I’d written for reformatting citations. In this entry I’ll talk about refactoring that emacs lisp code.

Ah, Refactoring. We’ll Need Some Tests…

Refactoring, by definition, is improving the internal structure of code without altering the external behaviour. Equally by definition, before you start you need thorough automated tests, because that’s how you tell that you haven’t altered the external behaviour.

I wrote the reformatting citations emacs lisp as an exploratory spike, looking up commands as I went, and manually testing the results. Time to get more rigorous. What is emacs lisp’s equivalent of JUnit, or MiniTest or RSpec?

ERT: Emacs Lisp Regression Testing

In JUnit you annotate test methods with “@Test”:

@Test
public void formatingRemoved {

In MiniTest, you start the method definitions with “test_”:

def test_formatting_removed

In ERT, where you would define a normal lisp function, with “defun”:

(defun remove-formatting ()
)

you can define a lisp test with “ert-deftest”:

(ert-deftest remove-formatting ()
)

Inside the test, you can call the function under test and compare the actual and expected results with the “should” macro. For instance, given a function that takes a string and removes some formatting (using replace-regexp-in-string):

(defun remove-formatting (string)
(replace-regexp-in-string "^> " ""
(replace-regexp-in-string "\s*<br/?>" "" string)))

We could write a test that checks that it does what it says:

(ert-deftest remove-formatting ()
(should (string= (remove-formatting "> Elþeodigra eard<br/>")
"Elþeodigra eard")))

To run all the tests, we can type:

M-x ert RET RET

The second RET accepts the default, t, and runs all tests. In this case there’s only one: if we have a large suite and we only want to run a subset, say those with “formatting” in the test name, we would type instead:

M-x ert RET "*formatting*" RET

In either case, another buffer is opened up with the results, for instance

Selector: t
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2015-02-02 16:03:15-0500
Finished.
Finished at:  2015-02-02 16:03:15-0500

.

And yes, as you would expect from other languages, that’s a dot per passing test, and an F for any failing test, so with twelve tests and two failures you might instead see:

..FF........

In the ERT results buffer, with the cursor on any . or F test result you have several options, including:

. ;; jump to that test’s source code
l ;; list the assertions run by the test
h ;; see the description string for the test, if any
b ;; view backtrace
r ;; re-run this test

As Nic Ferrier notes, the runner doesn’t automatically recognize when you delete a test, and you need to delete it in the ERT results buffer. I had trouble with it recognizing added tests as well, and ended up closing and restarting emacs to make sure it picked up the latest list. This is obviously not scalable. There’s a separate tool called ert-runner which fixes this, as I discover here.

Additional Complications

If we were testing code that took a string and reformatted it, like the example above, we could just write (should (string= examples for all the edge cases we could think of using what we already know, and we’d be set. Unfortunately, the code to be put under test also makes changes to the buffer, varies its behaviour depending on whether a region is selected when it is called, and modifies the cursor position. How do we handle that?

The with-temp-buffer macro saves the current buffer, creates an (empty) temporary buffer, marks it current, uses it inside the body of the macro, and on exit switches back to the previous current buffer. You can return the contents of the temporary buffer by using (buffer-string) as the last form, and the position of the cursor within the temporary buffer by using (point) as the last form. This lets us write tests for (begin-end-quote) when a region is not selected like so:

(ert-deftest test-begin-end-quote-new-content ()
"Tests begin-end-quote without preselected text string"
(should (string= (with-temp-buffer
(begin-end-quote)
(buffer-string))
"#+begin_quote\n\n#+end_quote\n")))

(ert-deftest test-begin-end-quote-new-point ()
"Tests begin-end-quote without preselected text cursor position"
(should (equal (with-temp-buffer
(begin-end-quote)
(point))
(length "#+begin_quote\n\n"))))

There’s a further complication for the case where a region is selected before calling it. We can include text in the temporary buffer before calling begin-end-quote by using insert, and then (set-mark .N.) to set the mark at the nth character, and then either goto-char .N. to the select the region from the first mark up to character n, or just do end-of-buffer to select the region to the end of the buffer. So to insert the text “> Dear Sir, your astonishment’s odd;\n” into the temporary buffer and select the whole region, we could do the following:

(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(end-of-buffer)

With that extra information, the tests of behaviour with a selected region become simple too:

(ert-deftest test-begin-end-quote-region ()
"Tests begin-end-quote with selected region"
(should (string= (with-temp-buffer
(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(buffer-string))
"#+begin_quote\n Dear Sir, your astonishment’s odd;\n#+end_quote\n")))

The commit with the full set of twelve tests is here.

As well as adding tests this commit makes a code change, because in the three weeks since writing it I have discovered that the archive files don’t always have exactly two spaces between the end of the text and the “<br/>”, so I wrote tests to expose that (so that two of them were failing, as in the example above), and then changed the regexp so that they passed.

Now that we’ve got full automated tests, we can start refactoring the code.

Emacs Org-mode: Links and Exported Html

If you have an archive of files in org-mode and you want to link between them, say from today’s entry to the entry of 9 March 2013, you have several options, as laid out here:

http://orgmode.org/manual/Search-options.html

The Simplest Case, for .org

The simplest is to provide the text of the header in the link, like so: so:

[[file:2013.org::Saturday 9 March 2013][9 March 2013]]

which, on typing the final closing square bracket, will collapse on screen to “9 March 2013”, and when you’re on the link and type “C-c C-o”, it will open the “2013.org” file at the header “Saturday 9 March 2013”. If you find you’ve made an error in the link target or title, typing “C-c C-l” will let you edit it.

http://orgmode.org/manual/Handling-links.html

And this works perfectly, until you export it to html, and then, of course, it doesn’t.

The Extra Step, for .html

To get a link also to work in html, you need to set a custom_id on the header, which you do like this:

*** Saturday 9 March 2013
:PROPERTIES:
:CUSTOM_ID: 20130309
:END:

or, less manually, with the org-set-property command (keyboard shortcut C-c C-x p):

C-c C-x p RET CUSTOM_ID RET 20130309

This will also attach an id attribute to the header element in the exported html, so in both .org and .html it is recognized as #20130309, and if you then change the link to

[[file:2013.org::#20130309][9 March 2013]]

it will work as before in the *.org file, and also in the exported *.html file.

The Final Step

Having done this a few times, you may get tired of typing “CUSTOM_ID” each time, and build a function that lets you just type the id value:

(defun cid (custom-id)
(interactive "MCUSTOM_ID: ")
(org-set-property "CUSTOM_ID" custom-id))

after which you can type, even more briefly:

M-x cid RET 20130309 RET

commit

Beginning Emacs Lisp

The Problem

A while back I converted an archive of non-code-related files to org-mode. The files had citations in a markdown-like format, so:

> ADA  <br/>
COUNTESS OF LOVELACE  <br/>
1815-1852  <br/>
Pioneer of Computing  <br/>
lived here

which, processed through Calibre, produced a nice readable pdf with blockquotes and linebreaks.

For a while after that, I wrote in org-mode and viewed the files within emacs, so I just indented quotations, like so:

.
    Quandunque i colli fanno più nera ombra,
    Sotto il bel verde la giovane donna
    Gli fa sparir, come pietra sott’ erba.

and then one day I used org-mode’s export-to-html functionality (C-c C-e h o) and I lost all the blockquoting and line-breaks. org-mode needs prose citations surrounded by

#+begin_quote
#+end_quote

to be rendered with <blockquotes> in export-to-html, and if you want it to respect line breaks as well, you need to use

#+begin_verse
#+end_verse

So. Going forward I needed to insert begin/end blocks for new citations, and I had a bunch of older citations I would need to reformat. Here’s what I build with emacs lisp to solve the problem.

The Solution

The Simplest Case

In a new file, about to add a quotation, I want a shortcut to add either #+begin_quote / #+end_quote or #+begin_verse / #+end_verse and put the cursor on the empty line between. This sounds like what yasnippets is designed for, but I wasn’t sure how that would work when I got to converting existing quotations, so I broke out my ~/.emacs.d/mods-org.el file and added two new shortcuts:

(global-set-key (kbd "M-s M-q")
(lambda()
(interactive)
(insert "#+begin_quote")
(newline)
(newline)
(insert "#+end_quote")
(newline)
(previous-line)
(previous-line)))

(global-set-key (kbd "M-s M-v")
(lambda()
(interactive)
(insert "#+begin_verse")
(newline)
(newline)
(insert "#+end_verse")
(newline)
(previous-line)
(previous-line)))

This meets my simplest case requirement and is fairly self-explanatory. Going forward, I can type

M-s M-q ;; q for quote

to get

#+begin_quote
_
#+end_quote

and

M-s M-v ;; v for verse

to get

#+begin_verse
_
#+end_verse

commit

Narrowing the Scope

The fact that I put them in ~/.emacs.d/mods-org.el instead of ~/.emacs.d/key-bindings.el foreshadows the next step: they aren’t global key bindings, I’m only going to use them in org-mode, and in a ruby class they’d just be noise. So, since I’ve already got a hook for entering text-mode (which I’m also using for org-mode), let’s change them from global key bindings to key bindings which are added to org-mode specifically.

I’ve got an ~/.emacs.d/mode-hooks.el file which already has a custom text-mode-hook, so I can add two lines to that:

(defun my-text-mode-hook ()
…
(define-key org-mode-map (kbd "M-s M-q") ‘begin-end-quote)
(define-key org-mode-map (kbd "M-s M-v") ‘begin-end-verse)
)

(add-hook ‘text-mode-hook ‘my-text-mode-hook)

And change the functions, in the ~/.emacs.d/mods-org.el file, from anonymous functions in the global-set-key blocks to the named functions we have just referenced:

(defun begin-end-quote ()
(interactive)
(insert "#+begin_quote")
(newline)
(newline)
(insert "#+end_quote")
(newline)
(previous-line)
(previous-line))

(defun begin-end-verse ()
(interactive)
(insert "#+begin_verse")
(newline)
(newline)
(insert "#+end_verse")
(newline)
(previous-line)
(previous-line))

Now if we’re in org-mode, the key-bindings do what we expect, but if we’re in ruby-mode and type them, or type (C-h k) to get the definition of a key binding and then type (M-s M-v), we get

M-s M-v is undefined

commit

Reformat Existing

That handles the going forward case, but doesn’t handle the case where I’m in an older file and want to reformat an existing quotation. To handle both, I’d like to check when I use the shortcut to see if I’ve got a selected a region or not. If I haven’t, it’s the going forward case, and I should do what I was doing before, but if I have, then presumably I want to put the #+begin and #+end blocks around the selected region.

We can tell this because emacs lisp gives us a function use-region-p which returns true if there is a region selected. The most basic if block in emacs lisp looks like this:

(if (condition)
(do-true-thing)
(do-false-thing))

so in our case we have:

(if (use-region-p)
(begin-end-quote-for-region)
(begin-end-quote-for-new))

and begin-end-quote-for-new is the old begin-end-quote method, and begin-end-quote-for-region looks like this:

(defun begin-end-quote-for-region ()
(interactive)
(insert "#+end_quote")
(newline)
(goto-char (region-beginning))
(insert "#+begin_quote")
(newline))

An extra bit of inwardness here is that the cursor starts at the end of the selected region, so we can just insert “#+end_quote” and it will show up after the end, and (region-beginning) and (region-end) hold the beginning and end of the selected region, so (goto-char (region-beginning)) gets us back to the beginning so we can insert “#+begin_quote” before it.

commit

This gets us to the point where if we’d selected the first quotation and hit M-s M-v, we’d end up with

#+begin_verse
> ADA  <br/>
COUNTESS OF LOVELACE  <br/>
1815-1852  <br/>
Pioneer of Computing  <br/>
lived here
#+end_verse

which is a definite improvement, but it still has the old formatting codes. Can we get rid of those?

Remove Old Formatting

First off, we only want to do this in the reformat existing case. That’s fine, those two methods (begin-end-quote-for-region and begin-end-verse-for-region) are already separate, so in each of those methods include a (remove-old-formatting) method.

What we want to do in pseudo-code is take the selected region and apply

s/^> //
s/  <br/>$//

to it. (We only need the second transformation for verse, not quotes, and if these got any more complicated we might want two separate methods, but we can leave them in one for now.)

setq defines a variable.

filter-buffer-substring grabs the text from arg1 to arg2 (and we’re using (region-beginning) and (region-end) which return the start and end of the selected region), and with the optional third argument of t deletes the text after copying it.

After that, we’ve got the contents of the selected region in a variable, “in”, and we can run replace-regexp-in-string on it, taking as arguments search-value, replace-value, and string-to-search in, and using setq to define to variable we’re storing the result in.

Once we’ve made all the changes we need, we use insert the finally modified string back into the buffer.

(defun remove-old-formatting ()
(setq in (filter-buffer-substring (region-beginning) (region-end) t))
(setq out (replace-regexp-in-string "^> " "" in))
(setq out2 (replace-regexp-in-string " <br/>$" "" out))
(insert out2)
)

commit

And at the end of that, we get:

#+begin_verse
ADA
COUNTESS OF LOVELACE
1815-1852
Pioneer of Computing
lived here
#+end_verse

Which is very nearly there, but not indented. How about as a last step we indent it, so if we’re viewing it in org-mode it still looks like a blockquote?

Indenting

Let’s put this in our remove-old-formatting method, because again it’s something that we’ll only want in the reformat existing case. Given that it’s no longer just removing old formatting, let’s change the method name to fix-old-formatting, and to keep things at the same level of abstraction, let’s put the old remove-old-formatting lines in a new method called remove-old–formatting-code and add a method indent-if-not-indented. So we have:

(defun fix-old-formatting ()
(remove-old-formatting-code)
(indent-if-not-indented)
)

(defun remove-old-formatting-code ()
(setq in (filter-buffer-substring (region-beginning) (region-end) t))
(setq out (replace-regexp-in-string "^> " "" in))
(setq out2 (replace-regexp-in-string " <br/>$" "" out))
(insert out2)
)

Remember that we have some existing citations in the form

> ADA  <br/>

and some already indented as

.
    Quandunque i colli fanno più nera ombra,

Further, some of the already indented ones have multiple layers of indentation, and setting a single indentation would break that. So while indent-region itself is simple enough, once again using (region-beginning) and (region-end) to give us the selected region, and the third argument for the number of columns to indent:

(indent-region (region-beginning) (region-end) 4)

we need to make it conditional on it not already being indented, so we end up with:

(defun indent-if-not-indented ()
(setq firstFour (filter-buffer-substring (region-beginning) (+ (region-beginning) 4)))
(if (not (string= firstFour " "))
(indent-region (region-beginning) (region-end) 4)
)
)

using filter-buffer-substring again to grab the first four characters of the region (without the optional third argument so we don’t delete it), and if they aren’t spaces, do the indent. If they are, it’s one of the newer existing quotations and we should leave it as it is, as one of them for instance was pseudo-code

.
    count = 500 
    day.each do
      wrote_words?(count) ? 
        count += 100 : 
        count -= 100
      count = 100 if count < 100
      count = 1500 if count > 1500
    end

and not doing that check would have clobbered all the internal indenting.

commit

With that, we finally get the desired end result for the older existing quotations:

#+begin_verse
    ADA
    COUNTESS OF LOVELACE
    1815-1852
    Pioneer of Computing
    lived here
#+end_verse

without clobbering existing indentation for the more recent existing quotations:

#+begin_verse
    count = 500
    day.each do
      wrote_words?(count) ?
        count += 100 :
        count -= 100
      count = 100 if count < 100
      count = 1500 if count > 1500
    end
#+end_verse

End and Afterthoughts

The current state of my ~/.emacs.d/ is here. It is very much a work in progress. Also, having just looked at Sacha Chua’s more literate emacs config using org-babel, I’m quite tempted to try that out, instead of composing the two or three additional explanatory/introductory blog posts that occurred to me would be helpful as I was writing this up.

Notes and Quotes from Conference Talks

What did we do before Confreaks let us catch up on all the conference talks? Here are notes on some that resonated with me recently.

TDD for Your Soul: Virtue and Web Development with Abraham Sangha at Madison+ Ruby

… more philosophy than tech, but thought-provoking.

citing Alasdair McIntyre, “After Virtue”:

“Who am I?

Who ought I to become?

How ought I to get there?”

citing Ralph Ellison:

“The blues is an impulse to keep the painful details and episodes of a brutal experience alive in one’s aching consciousness, to finger its jagged grain, and to transcend it, not by the consolation of philosophy but by squeezing from it a near-tragic, near-comic lyricism.”

Abraham Sangha:

“We’re inviting criticism of ourselves, we’re inviting evaluation of our weaknesses, by TDDing our soul, by writing these tests, seeing where we fail, and trying to implement habits to address these failures: that can be a shameful process. … If you don’t believe you’re good enough, you’re stuck, then why would you invite more pain into your life through this process, so why not just skip it. … But there’s a possibility that pursuing this will actually give us a sense of buoyancy.”

Alchemy and the Art of Software Development with Coraline Ada Ehmke at Madison+ Ruby

Going back in pattern languages before Christopher Alexander to Gottfried Wilhelm von Leibniz (1646-1716): “every complex idea is composed of sets of simpler ideas.”

“alchemy is about transforming base matter, like the stuff that we’re made of, into something more closely approaching divine perfection. That ‘lead into gold’ nonsense was actually part of the pitch that alchemists made to the VCs of their era. They called them royalty, I’m not quite sure why. And this idea that ‘I can take base metal and transmute it into gold? Here: I’ll give you all this money for your metaphysical experiments. I don’t care.’ So they were pretty smart.”

“The Divine Pymander, ascribed to Hermes Mercurius Trismegistus. … It contains seventeen tracts, which talk about things like divinity, humanity, idealism, even monads.”

“we impose our will on the universe, we create a structure that we want to impose on chaos, and the manifestation of that begins in harmony, but slides towards disharmony, because every object in every system is corruptible. … the code is corruptible.”

“all that is apparent, is what was generated or made. … the system does not contain information about the ideals that led to its creation. Unless we are very deliberate about recording our intention and our design, that information is lost, and all we are left with is a system that we don’t understand any more.”

“Christopher Alexander the architect said that a builder should start with the roughest sketch of a building, and by processes that are known to the brain through the pattern language, execute the construction of the building. All things that are are but imitations of truth. The systems we build are reflections of the world. Software system is not and cannot be a single source of truth.”

“Really, alchemy and software development are about identifying ideals, identifying particulars, creating taxonomies, studying them, creating a system for classifying every single thing in a limited or expansive universe.”

“The Divine Pymander ends with this: ‘The Image shall become thy Guide, because Sight hath this peculiar charm, it holdeth fast and draweth unto it those who succeed in opening their eyes.’ I would translate this as ‘The metaphors guide us. Information is there, and ideas want to be recognized.’ I think that the metaphor of alchemy holds true with what we do. Just like alchemy, software development is an inquiry into the essential nature of reality. We break it down, we salve it coagula, we break down complex things into simpler things, we recombine them in novel ways, and we produce an effect on the world, on ourselves.”

“Let’s … dive into disciplines that we have no idea what they even mean, explore them, mine them, find tenets and metaphors, tools that we can use in problem solving, that we can apply to enrich our art, our science, our own magnum opus. Maybe when we do that, we can find that the raw stuff of digital creation can be transmuted into something that more closely approaches perfection.”

Aesthetics and the Evolution of Code: Coraline Ada Ehmke, Nickle City Ruby 2014

Aesthetics of code: correctness, performance, conciseness, readability.

So how do we measure elegance? How about a graph with four lines/axes spreading from the origin, correctness “north”, performance “south”, conciseness “west”, readability “east”. If you can measure and grade those four aesthetics on a numeric scale, then the most elegant code will cover the largest space on the graph

Why does it matter? Einstein said:

“After a certain level of technological skill is achieved, science and art tend to coalesce in aesthetic plasticity and form. There greater scientists are artists as well.”

Eric Stiens’s Nickel City Ruby Conference 2014 talk “How to be an Awesome Junior Developer (and why to hire them)”

On Mentoring:

Don’t hire a junior developer you can’t mentor. It’s not fair to them. It’s not fair to your team. Everybody loses.

Sarah Mei’s Nickel City Ruby Conference 2014 talk “Keynote: Multitudes”

Should you always use attr_reader to access an instance variable rather than accessing it directly, as a good object-oriented principle, because by only accessing it by sending a message you have created a seam which makes it easier to change anything about the implementation later? A good application developer (e.g. Sandi Metz) would very likely say yes. A good library developer (e.g. Jim Weirich) might say no, because adding attr_reader to a class makes data manipulation public, and once it’s public it has to be supported for ever, because people will use it. (And if you do change it, and you’re using semantic versioning, you have to change the major version.)

“So people who write gems have developed a system where they have a very minimal interface, with a very rich set of internal abstractions, which they can use while providing this very minimal external interface.

“So these are just two different approaches to programming [application style at one end, gem/library style at the other], and what we’re starting to see here is there is a spectrum of rubyists, and there is a spectrum of projects: people will write different code at different points of the spectrum at different times, and what the spectrum is measuring is the surface area of our interface. And it seems like a fairly simple distinction but it does produce a huge difference in code structure. And what it means is that sometimes someone who is good at one side of this spectrum will not automatically be good at the other side, right away. … Why does this matter? … Having these endpoints helps us define what else there is. For example, there’s a middle here, and that middle is suddenly quite obvious, actually, and that’s external APIs on Rails apps. Many Rails apps now need some kind of programmatic interface that is versioned, and minimal, because once it’s out there, and it’s got users, you’re stuck supporting it, forever. … And the reason people struggle with external APIs for Rails apps is that it is partially application development and it is partially gem-style development, and there aren’t very many people that are good at both.”

Exciting that Sarah Mei and Sandi Metz are writing a book: Practical Rails Programming.