Posts Tagged → emacs lisp

Automating Boilerplate in Org-mode Journalling

The Problem

My main org-mode files have a header structure like this:

* 2015
** March
*** Friday 6 March 2015
**** 10:00. Emacs coaching with Sacha :emacs:
<2015-03-06 10:00-11:00>
**** 23:37. Emacs progress note
[2015-03-06 23:37]
*** Saturday 7 March 2015
**** 12:55. Stratford King Lear in cinemas :wcss:
<2015-03-07 12:55-15:55>

The two things I do most often are start writing notes in the present, and set up scheduled events in the future. Scheduled events need active timestamps so that they show up in the agenda view:

Week-agenda (W10):
Monday 2 March 2015 W10
Tuesday 3 March 2015
Wednesday 4 March 2015
Thursday 5 March 2015
Friday 6 March 2015
10:00. Emacs Coaching with Sacha :emacs:
Saturday 7 March 2015
12:55. Stratford King Lear in cinemas :wcss:
Sunday 8 March 2015

[How to get the agenda view? Make sure the file you’re in is in org-agenda-files by pressing C-c [, then run org-agenda either by M-x org-agenda or the common (but not out-of-the-box) shortcut C-c a, then press a to get the agenda for the week.]

[Active timestamps are in < >; inactive timestamps are in [ ]. If we want to promote an inactive timestamp to be active and show up in the agenda view, we can run M-x org-toggle-timestamp-type from the inactive timestamp.]

[Start time is duplicated between header and timestamp deliberately: sometimes I review the file in org-mode outline with only the headers. If not for that use-case it would make sense to leave the start time out of the header.]

What I’d like to be able to do is open the file, and then with as little ceremony or setup as possible, start typing.

Adding a New Note

When I’ve already got a date header for the current date, this is trivially simple: I can put my cursor under that date and call M-x my note, having previously defined

(defun my/note ()
(interactive)
(insert "\n\n**** " (format-time-string "%H:%M") ". ")
(save-excursion
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n")))

… and it does pretty much what it says.

The values passed into format-time-string (%Y, %m, %d etc.) will be the values for the current date and time.

save-excursion means “do what’s inside here and then return the point (cursor) to where it was before the block was called”, so it fills in the timestamp and then returns the cursor to the end of the header line, ready for the title to be filled in.

Since after that I’d need to move the cursor back down to below the timestamp, I can be even lazier and prompt for the title and the tags (if any) and fill them in and move the cursor down to where the text should start:

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

If we were just setting one argument, say title, this would be simpler:

(defun my/note-2-eg (title)
(interactive "MTitle? ")

But since we want two, we need to tell interactive it’s getting a list, and then the arguments are set in order.

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))

Just as we can create an inactive timestamp by inserting a formatted date with [ ], we can add tags by inserting a string surrounded by : :. We don’t want to do this if the tag argument was left blank, of course, so we surround it with an unless:

(unless (string= tags "")
(insert " :" tags ":")
)

[Side-note: if you’re doing anything more complicated than this with strings, you probably want Magnar Sveen’s s.el (string manipulation library), which here would let you do (unless (s-blank? tags)… instead.]

At the end of which, because we took out the save-excursion, the cursor is below the timestamp and I’m ready to type.

All good. What if we don’t already have a date header for the current date? Can we auto-generate the date header?

Auto-generating Date Headers

Org-mode has a function called org-datetree-find-date-create. If you pass it in a list of numbers (the month, the day, and the year), and if your header structure is

* 2015
** 2015-03 March
*** 2015-03-06 Friday

then if you called that function passing in (3 7 2015), it would automatically add:

*** 2015-03-07 Saturday

For that matter, if you called it passing in (4 1 2015), it would automatically add

** 2015-04 April
*** 2015-04-01 Wednesday

If you call it passing in a date which is already there, it moves the cursor to that date. So, we could change the format of the org file headers, update our new note function to call the function, and be done.

(defun my/note-3 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

In the new bit:

(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)

format-time-string “%Y-%m-%d” will return “2015-03-07”, and org-date-to-gregorian will turn that into (3 7 2015), which is the format that org-datetree-find-date-create expects. Determining this involved looking at the source for org-datetree-find-date-create to see what arguments it expected (C-h f org-datetree-find-date-create takes you to a help buffer that links to the source file, in this case org-datetree.el; click on the link to go to the function definition) and a certain amount of trial and error. At one point, before org-date-to-gregorian, I had the also working but rather less clear:

(org-datetree-find-date-create (mapcar ‘string-to-number
(split-string (format-time-string "%m %d %Y"))))

org-end-of-subtree just takes the cursor to the bottom of the section for the date.

And that then works. What about adding new events in the future?

Adding a New Event

org-datetree-find-date-create makes it easier to fill in missing month and date headers to create a new future event:

(defun my/event (date end-time)
(interactive (list
(org-read-date)
(read-from-minibuffer "end time (e.g. 22:00)? ")))
(org-datetree-find-date-create (org-date-to-gregorian date))
(goto-char (line-end-position))
(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")
(save-excursion
(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n")))

There’s a problem I haven’t solved here, which is that org-read-date brings up the date prompt and lets you select a date, including a time or time range. If you select a time, it will be included in the date. If you select a time range, say 19:30-22:30, it ignores the time and the date object returned uses the current time. That’s not what we want.

So when the date prompt comes up:

Date+time [2015-03-07]: _ => <2015-03-07 Sat>

I can put in a new date and start time, say “2015-04-01 11:00”:

Date+time [2015-03-07]: 2015-04-01 11:00_ => <2015-04-01 Wed 11:00-14:00>

and then press enter and that gives me the date and the start time in a single string. We extract the start time and put it into the header like so:

(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")

Where split-string splits the date “2015-04-01 11:00” into a two-element list (“2015-04-01” “11:00”) {n}, and nth 1 returns the second element (list elements, here as elsewhere, are numbered starting from 0), or “11:00” {n}.

If the date had been “2015-04-01” without a start time, line 1 would have set start-time to nil, which would have blown up as an argument to insert (with a “Wrong type argument: char-or-string-p, nil”), so we check for a nil and set it to an empty string instead.

For the agenda view we’ll need the end-time as well, so we grab that as a second argument, and reassemble the date, say “2015-04-01 11:00”, and end-time, say “14:00”, into the active timestamp:

(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n"))

though here again, if we had just pressed enter when prompted for end time we would end up with an empty string and an invalid active timestamp, like “<2015-04-01->”, so check whether end-time has been set and build the timestamp string accordingly.

And since this is a future event and we’re probably only filling in the title, we’ll use save-excursion again to fill in the active timestamp and then go back and leave the cursor on the header to fill in the title.

The End

And we’re done. Or we would be, but I would really prefer the header to read “March” instead of “2015-03 March”, and “Saturday 7 March 2015” instead of “2015-03-07 Saturday”. We might need to come back to that.

Shortcuts to Default Directories

[Now largely historical. See ETA for achieving the same effect with bookmarks, and ETA 2 for achieving the same effect with Hydra, which feels better still.]

Yesterday’s Emacs coaching session with Sacha Chua included a sidebar on jumping to and setting default directories. Sacha’s .emacs.d uses registers for this, and her code sets defaults and org-refile targets.

I found later that I had six shortcuts I’d clean forgotten about, that set the default directory and open either dired or a specific file:

(global-set-key (kbd "C-c C-g C-c")
(lambda ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired ".")))

(global-set-key (kbd "C-c C-g C-h")
(lambda ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org")))

But a forgotten solution doesn’t count, and having to remember triple-decker key bindings doesn’t help either.

A general solution for the second problem is that an incomplete key binding followed by ? opens a help buffer with all the bindings starting with sequence. C-c C-g ? on my machine yielded:

Global Bindings Starting With C-c C-g:
key binding
— ——-

C-c C-g C-c ??
C-c C-g C-d ??
C-c C-g C-e to-english
C-c C-g C-f to-french
C-c C-g C-h ??
C-c C-g C-p ??
C-c C-g C-r ??
C-c C-g C-u ??
C-c C-g C-w ??

We can make this more useful by switching from defining the key binding functions anonymously inline to moving them out to their own named functions, thus:

(defun my/to-emacs ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired "."))

(defun my/to-today ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org"))

(global-set-key (kbd "C-c C-g C-c") ‘my/to-emacs)
(global-set-key (kbd "C-c C-g C-h") ‘my/to-today)

Which fixes those two ?? entries:

C-c C-g C-c my/to-emacs
C-c C-g C-h my/to-today

Another niggle is that I’ve overloaded C-c C-g across three different kinds of commands. Let’s make C-x j the shortcut specifically and only for jumping to a new default directory, and extract some duplicate methods while we’re at it. After:

(defun my/to-file (dir file)
(interactive)
(setq default-directory dir)
(find-file file))

(defun my/to-dir (dir)
(interactive)
(setq default-directory dir)
(dired "."))

(defun my/to-gesta-file (file)
(interactive)
(my/to-file "~/Dropbox/gesta/" file))

(defun my/to-emacs-config ()
(interactive)
(my/to-file "~/.emacs.d/" "sean.org"))

(defun my/to-autrui ()
(interactive)
(my/to-dir "~/code/autrui/"))

(defun my/to-gesta ()
(interactive)
(my/to-dir "~/Dropbox/gesta/"))

(defun my/to-today ()
(interactive)
(my/to-gesta-file "2015.org"))

(defun my/to-readings ()
(interactive)
(my/to-gesta-file "readings.org"))

(defun my/to-writings ()
(interactive)
(my/to-gesta-file "writings.org"))

(defun my/to-twc ()
(interactive)
(my/to-dir "~/Dropbox/gesta/twc/"))

(global-set-key (kbd "C-x j e") ‘my/to-emacs-config)
(global-set-key (kbd "C-x j a") ‘my/to-autrui)
(global-set-key (kbd "C-x j g") ‘my/to-gesta)
(global-set-key (kbd "C-x j h") ‘my/to-today)
(global-set-key (kbd "C-x j r") ‘my/to-readings)
(global-set-key (kbd "C-x j w") ‘my/to-writings)
(global-set-key (kbd "C-x j t") ‘my/to-twc)

C-x j ? then yields:

Global Bindings Starting With C-x j:
key binding
— ——-

C-x j a my/to-autrui
C-x j e my/to-emacs-config
C-x j g my/to-gesta
C-x j h my/to-today
C-x j r my/to-readings
C-x j t my/to-twc
C-x j w my/to-writings

Which is an improvement, but the “Bindings Starting With” help menu is not interactive: we still have to either remember the triple-decker key binding, or type C-x j ? to find the list and then type C-x j h (say) to get the specific shortcut we want. It would be nice if we could over-ride C-x j ? and have it prompt us for one more character to select which of the seven jumps we wanted.

Something like this, in fact:

(defun my/pick-destination (pick)
(interactive "ce = ~/.emacs.d/sean.org a = ~/code/autrui/ g = ~/Dropbox/gesta/ h = …/2015.org r = …/readings.org w = …/writings.org t = …/twc/ ? ")
(case pick
(?e (my/to-emacs-config))
(?a (my/to-autrui))
(?g (my/to-gesta))
(?h (my/to-today))
(?r (my/to-readings))
(?w (my/to-writings))
(?t (my/to-twc))))

(global-set-key (kbd "C-x j ?") ‘my/pick-destination)

interactive "c… takes a single character (see further options here), so the conditional is on a character, which in Emacs Lisp is represented by a ? followed by the character {n}. If there were a single conditional we might use (when (char-equal ?e pick) {n}, but since there are seven of them, we look instead for Emacs Lisp’s equivalent of a switch statement, and find it in case (an alias for cl-case).

So what this does, when you type C-x j ?, is provide a prompt of options in the echo area at the bottom of the screen, and if you type one of the significant letters, it uses that shortcut to default.

Purists would probably prefer to bind ’my/pick-destination to something other than C-x j ? (C-x j j say), so that if we ever bound other commands to something starting with C-x j we would still be able to discover them with C-x j ?. It’s also easier to type because it doesn’t need the shift key for the third element. Having demonstrated that we could over-ride C-x j ? if we wanted to, I’m probably going to side with the purists on this one.

And we’re done. Commit.

ETA: Easier Ways To Do It

That works, but since (see Sacha’s comment below) there are always easier ways to do it:

Simpler Code

find-file will either open a file or open dired if it is passed a directory instead of a file, and opening a file or directory will change the default directory. So we can replace all the extracted ’my/to-file and ’my/to-dir methods with simple find-file calls:

(defun my/to-emacs-config ()
(interactive)
(find-file "~/.emacs.d/sean.org"))

(defun my/to-autrui ()
(interactive)
(find-file "~/code/autrui/"))

… etc …

Using Bookmarks (No Custom Code, Even Simpler)

While ’my/pick-destination gets around the non-interactive nature of ? (minibuffer-completion-help, e.g. C-c C-g ? for list of completions to C-c C-g), using straight-up bookmarks lets you do that without custom code:

C-x r m {name} RET ;; bookmark-set ;; sets a {named} bookmark at current location
C-x r b {name} RET ;; bookmark-jump ;; jump to {named} bookmark
C-x r l ;; list-bookmarks ;; list all bookmarks

So having set bookmarks in the same seven places, either files or directories, we could list-bookmarks and get

% Bookmark File
autrui ~/code/autrui/
emacs ~/.emacs.d/sean.org
gesta ~/Dropbox/gesta/
now ~/Dropbox/gesta/2015.org
readings ~/Dropbox/gesta/readings.org
twc ~/Dropbox/gesta/twc/
writings ~/Dropbox/gesta/writings.org

And this list is interactive. Or we could call C-x r b (bookmark-jump) and start typing the destination we want. (If we get bored of typing, we can reduce the bookmark names to single characters, remembering that C-x r l (list-bookmarks) will give us the translation table if we forget, and then we’re back to being five keystrokes away from any bookmark, without having to add any extra code.)

Using Bookmark+ (And Describing and Customizing Faces)

If we want to be able to do more advanced things with bookmarks – tag them, annotate them, rename them, run dired-like commands on the bookmark list – we can grab the Bookmark+ package and (require ’bookmark+) in our .emacs. (Having done that, if we press e by a line in the bookmark list, for instance, we can edit the lisp record for the bookmark, to rename it or change the destination or see how many times it has been used.)

One problem I had with bookmark+ is that the bookmark list was displaying illegibly in dark blue on a black background. To fix this, I needed to move the cursor over one of the dark blue on black bookmark names and type M-x describe-face, and it reported the face (Emacs-speak for style {n}) of the character, in this case Describe face (default `bmkp-local-file-without-region'):. Pressing enter took me to a buffer which described the face and provided a customize this face link at the end of the first line. I followed (pressed enter) that link to get to a customize face buffer which let me change the styling for that element. On the line:

[X] Foreground: blue [ Choose ] (sample)

I followed (pressed enter on) Choose, it opened a buffer of colour options, I scrolled up to a more visible one, pressed enter again, and got back to the customize face buffer, then went up to “Apply and Save” link and pressed enter again there. Going back to the bookmark list, the bookmarks were visible. The change is preserved for future sessions in the init.el file:

(custom-set-faces
;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won’t work right.
‘(bmkp-local-file-without-region ((t (:foreground "green")))))

ETA 2: Using Hydra Instead

Heikki Lehvaslaiho suggested using hydra instead, and as the bookmark solution required a couple of extra keystrokes and I’d been curious about hydra anyway, I thought I’d give it a go.

(require ‘hydra)
(global-set-key
(kbd "C-c j")
(defhydra hydra-jump (:color blue)
"jump"
("e" (find-file "~/.emacs.d/sean.org") ".emacs.d")
("c" (find-file "~/.emacs.d/Cask") "Cask")

("a" (find-file "~/code/autrui/") "autrui")
("h" (find-file "~/Dropbox/gesta/2015.org") "hodie")
("r" (find-file "~/Dropbox/gesta/readings.org") "readings")
("w" (find-file "~/Dropbox/gesta/writings.org") "writings")
("t" (find-file "~/Dropbox/gesta/twc/") "twc")))

I like it. We’re back down to three keystrokes from five and, since we’re providing labels for the commands, we also get the descriptive and interactive and self-updating index which was the big advantage of the bookmark route. If you press C-c j and don’t immediately carry on, it prompts:

jump: [e]: .emacs.d, [c]: Cask, [a]: autrui, [h]: hodie, [r]: readings, [w]: writings, [t]: twc.

Much better. Thanks for the prompt, Heikki!

Refactoring “Beginning Emacs Lisp”: II

Now that we have test coverage and a test runner, we can refactor the begin/end code.

First off, let’s namespace it. Instead of defun begin-end-quote (), let’s make it defun my/begin-end-quote ().

Why? There is another library with a function called begin-end-quote: I’m not using it, but if I were, after namespacing I wouldn’t have to worry about anything breaking because of the name collision.

The coding conventions suggest namespacing packages for widespread use with -, not / (q.v. typopunct-mode ’s typopunct-insert-quotation-mark), but namespacing “less formal” code with / seems a useful distinction. Sacha Chua suggested a further convention for Emacs config files of my/ instead of (say) sean/, since .emacs.d code is so often copied and borrowed and it would make it look more consistent after mixing code from multiple sources.

Run the tests: they still pass. Hurrah.

Almost all of the code between begin-end-verse and begin-end-quote is the same. I made them separate to begin with because I thought they would be more different, but they aren’t. So we can make everything but the outermost function take an argument of “quote” or “verse” and remove the duplication. (This will also let me quickly add variants like #+begin_src emacs-lisp and #+begin_example, which I’ve already needed while writing this blog.)

Again, run the tests, they still pass. Commit with this change.

Further quick points: newline does insert a carriage return, but insert “\n” would do the same thing, and feels more intuitive. Likewise, while previous-line and next-line do what they say, the more usual way would be to run forward-line with a positive or negative integer argument.

There is also duplication between my/begin-end-selected-region and my/begin-end-no-selected-region: they both print the #+begin and #+end tags, and if there’s a selected region it is reformatted, and if there isn’t, after printing the tags it moves the cursor back up between the tags. We can collapse the two methods into a single method with two conditionals.

(defun my/begin-end (variant)
(interactive)
(let ((cited-string "\n"))
(when (use-region-p)
(setq cited-string
(my/remove-old-citation-formatting (buffer-substring-no-properties (region-beginning) (region-end))))
(delete-region (region-beginning) (region-end)))
(insert "#+begin_" variant "\n"
cited-string
"#+end_" variant "\n"))
(unless (use-region-p)
(forward-line -2)))

The simpler conditional is at lines 11-12: (use-region-p) returns true if there is a region selected, and if there isn’t, it moves the cursor up two lines.

The one in lines 3-10 is more complicated.

In lines 3-7, we’re using a let statement to set cited-string initially to \n, which is what it should be if there’s no selection, and then we’re checking to see if there is a selected region: if there is, we’re resetting cited-string to the value of the selected region with the old formatting code stripped out, and then we’re deleting the selected region.

At the beginning of line 8, then, we’ve deleted the selected region if there was one, and until the end of the body of the let at the end of line 10, cited-string contains either a return character (if there was no selection) or the reformatted contents of the selected region (if there was a selection). Because of that, we can insert it between the tags either way.

As part of this, we also modified the my/fix-old-formatting method from one that dealt with the selected region in place to one which took a string and returned a modified string. All the selected region manipulation now takes place in the my/begin-end function.

Re-run the tests: they still pass. Excellent. Commit.

At this point, the refactoring is done, but it is now trivially easy to add new tests and implementations and key-bindings for #+begin_example / #+end_example and #+begin_src emacs-lisp / #+end_src.

The test for #+begin_src emacs-lisp breaks because for _src, the beginning tag and the end tag are different. The simplest way to get this passing is to pass in two arguments to my/begin-end, which for all but src will be identical, like so:

(defun my/begin-end-example ()
(interactive)
(my/begin-end "example" "example"))

(defun my/begin-end-src-emacs-lisp ()
(interactive)
(my/begin-end "src emacs-lisp" "src"))

(defun my/begin-end (begin-tag end-tag)
…
(insert "#+begin_" begin-tag "\n"
cited-string
"#+end_" end-tag "\n"))

At this point, the code does everything we need, the tests pass, and we drove out the new functionality we needed with tests. We’re done. Final commit.

Emacs Lisp: Adding Tests: ert-runner and overseer

In the previous Adding Tests post I found that running M-x ert on its own did not always pick up added or removed tests, but that a test running tool called ert-runner might fix this. It does, and adding another tool called overseer makes it easy to run tests or subsets of tests without leaving Emacs.

Here’s how I got there:

  1. Install Cask to manage project dependencies, as ert-runner uses it to run the tests from the command-line. I found that the default brew install cask didn’t set up ~/.cask/cask.el properly, but the longer curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python did.
    • After that, the usage instructions mostly worked out of the tin.
      • Calling cask init --dev sets up a Cask file (think Gemfile from ruby projects) with a block of development dependencies. We aren’t using ecukes or el-mock yet, and we’ll add overseer.
      • I also took out the (package-file "TODO") line, as it stopped cask install from running.
      • I had borrowed package-loading from bbatsov’s excellent Prelude: for now I’ve commented that out and put my Emacs package dependencies into the Cask file.
      • Running cask install puts all the packages in a .cask directory, so you might want to add that to .gitignore.
      • commit for adding Cask
  2. With ert-runner installed by Cask, from the command line, run cask exec ert-runner init. This creates a test directory and an empty test/test-helper.el file. Previously, our tests for the begin/end functionality were in the same file as the code: for ert-runner to be able to find them, they need to be in the test directory, in a file ending -test.el. So we move the tests from mods-org.el to test/mods-org-test.el.
    • The newbie / coming from Ruby mistake that I made at this point was to go to the command line and run cask exec ert-runner, expecting it to pick up the files automatically. This of course fails. We need to put a (provide 'mods-org) at the bottom of the mods-org.el file, and a (load-file "mods-org.el") at the top of test/mods-org-test.el.
    • Having done that, we can run the tests, and they all pass. Further, if some of them fail, we can do the usual thing: comment out all but one of them and re-run and, unlike with M-x ert, it picks up the changes and just runs a single test.
    • commit for reorganizing tests
  3. When going out to the command line or popping open a shell to run cask exec ert-runner every time proves irksome, we can take advantage of the fact that we also added overseer, which gives us some handy shortcuts. With a single test file so far, I have been using C-c , b to run all the tests from the buffer.

Thanks to Sacha Chua for unblocking me at one point, and sharing a draft post which takes things further into continuous integration and test coverage tools.

Refactoring “Beginning Emacs Lisp”: I: Adding Tests

On Friday I sat down with Sacha Chua for some emacs coaching. We talked about org-mode and about the emacs lisp I’d written for reformatting citations. In this entry I’ll talk about refactoring that emacs lisp code.

Ah, Refactoring. We’ll Need Some Tests…

Refactoring, by definition, is improving the internal structure of code without altering the external behaviour. Equally by definition, before you start you need thorough automated tests, because that’s how you tell that you haven’t altered the external behaviour.

I wrote the reformatting citations emacs lisp as an exploratory spike, looking up commands as I went, and manually testing the results. Time to get more rigorous. What is emacs lisp’s equivalent of JUnit, or MiniTest or RSpec?

ERT: Emacs Lisp Regression Testing

In JUnit you annotate test methods with “@Test”:

@Test
public void formatingRemoved {

In MiniTest, you start the method definitions with “test_”:

def test_formatting_removed

In ERT, where you would define a normal lisp function, with “defun”:

(defun remove-formatting ()
)

you can define a lisp test with “ert-deftest”:

(ert-deftest remove-formatting ()
)

Inside the test, you can call the function under test and compare the actual and expected results with the “should” macro. For instance, given a function that takes a string and removes some formatting (using replace-regexp-in-string):

(defun remove-formatting (string)
(replace-regexp-in-string "^> " ""
(replace-regexp-in-string "\s*<br/?>" "" string)))

We could write a test that checks that it does what it says:

(ert-deftest remove-formatting ()
(should (string= (remove-formatting "> Elþeodigra eard<br/>")
"Elþeodigra eard")))

To run all the tests, we can type:

M-x ert RET RET

The second RET accepts the default, t, and runs all tests. In this case there’s only one: if we have a large suite and we only want to run a subset, say those with “formatting” in the test name, we would type instead:

M-x ert RET "*formatting*" RET

In either case, another buffer is opened up with the results, for instance

Selector: t
Passed:  1
Failed:  0
Skipped: 0
Total:   1/1

Started at:   2015-02-02 16:03:15-0500
Finished.
Finished at:  2015-02-02 16:03:15-0500

.

And yes, as you would expect from other languages, that’s a dot per passing test, and an F for any failing test, so with twelve tests and two failures you might instead see:

..FF........

In the ERT results buffer, with the cursor on any . or F test result you have several options, including:

. ;; jump to that test’s source code
l ;; list the assertions run by the test
h ;; see the description string for the test, if any
b ;; view backtrace
r ;; re-run this test

As Nic Ferrier notes, the runner doesn’t automatically recognize when you delete a test, and you need to delete it in the ERT results buffer. I had trouble with it recognizing added tests as well, and ended up closing and restarting emacs to make sure it picked up the latest list. This is obviously not scalable. There’s a separate tool called ert-runner which fixes this, as I discover here.

Additional Complications

If we were testing code that took a string and reformatted it, like the example above, we could just write (should (string= examples for all the edge cases we could think of using what we already know, and we’d be set. Unfortunately, the code to be put under test also makes changes to the buffer, varies its behaviour depending on whether a region is selected when it is called, and modifies the cursor position. How do we handle that?

The with-temp-buffer macro saves the current buffer, creates an (empty) temporary buffer, marks it current, uses it inside the body of the macro, and on exit switches back to the previous current buffer. You can return the contents of the temporary buffer by using (buffer-string) as the last form, and the position of the cursor within the temporary buffer by using (point) as the last form. This lets us write tests for (begin-end-quote) when a region is not selected like so:

(ert-deftest test-begin-end-quote-new-content ()
"Tests begin-end-quote without preselected text string"
(should (string= (with-temp-buffer
(begin-end-quote)
(buffer-string))
"#+begin_quote\n\n#+end_quote\n")))

(ert-deftest test-begin-end-quote-new-point ()
"Tests begin-end-quote without preselected text cursor position"
(should (equal (with-temp-buffer
(begin-end-quote)
(point))
(length "#+begin_quote\n\n"))))

There’s a further complication for the case where a region is selected before calling it. We can include text in the temporary buffer before calling begin-end-quote by using insert, and then (set-mark .N.) to set the mark at the nth character, and then either goto-char .N. to the select the region from the first mark up to character n, or just do end-of-buffer to select the region to the end of the buffer. So to insert the text “> Dear Sir, your astonishment’s odd;\n” into the temporary buffer and select the whole region, we could do the following:

(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(end-of-buffer)

With that extra information, the tests of behaviour with a selected region become simple too:

(ert-deftest test-begin-end-quote-region ()
"Tests begin-end-quote with selected region"
(should (string= (with-temp-buffer
(insert "> Dear Sir, your astonishment’s odd;\n")
(goto-char (point-min))
(set-mark-command nil)
(goto-char (point-max))
(transient-mark-mode 1)
(buffer-string))
"#+begin_quote\n Dear Sir, your astonishment’s odd;\n#+end_quote\n")))

The commit with the full set of twelve tests is here.

As well as adding tests this commit makes a code change, because in the three weeks since writing it I have discovered that the archive files don’t always have exactly two spaces between the end of the text and the “<br/>”, so I wrote tests to expose that (so that two of them were failing, as in the example above), and then changed the regexp so that they passed.

Now that we’ve got full automated tests, we can start refactoring the code.

Emacs Org-mode: Links and Exported Html

If you have an archive of files in org-mode and you want to link between them, say from today’s entry to the entry of 9 March 2013, you have several options, as laid out here:

http://orgmode.org/manual/Search-options.html

The Simplest Case, for .org

The simplest is to provide the text of the header in the link, like so: so:

[[file:2013.org::Saturday 9 March 2013][9 March 2013]]

which, on typing the final closing square bracket, will collapse on screen to “9 March 2013”, and when you’re on the link and type “C-c C-o”, it will open the “2013.org” file at the header “Saturday 9 March 2013”. If you find you’ve made an error in the link target or title, typing “C-c C-l” will let you edit it.

http://orgmode.org/manual/Handling-links.html

And this works perfectly, until you export it to html, and then, of course, it doesn’t.

The Extra Step, for .html

To get a link also to work in html, you need to set a custom_id on the header, which you do like this:

*** Saturday 9 March 2013
:PROPERTIES:
:CUSTOM_ID: 20130309
:END:

or, less manually, with the org-set-property command (keyboard shortcut C-c C-x p):

C-c C-x p RET CUSTOM_ID RET 20130309

This will also attach an id attribute to the header element in the exported html, so in both .org and .html it is recognized as #20130309, and if you then change the link to

[[file:2013.org::#20130309][9 March 2013]]

it will work as before in the *.org file, and also in the exported *.html file.

The Final Step

Having done this a few times, you may get tired of typing “CUSTOM_ID” each time, and build a function that lets you just type the id value:

(defun cid (custom-id)
(interactive "MCUSTOM_ID: ")
(org-set-property "CUSTOM_ID" custom-id))

after which you can type, even more briefly:

M-x cid RET 20130309 RET

commit

Beginning Emacs Lisp

The Problem

A while back I converted an archive of non-code-related files to org-mode. The files had citations in a markdown-like format, so: 

> ADA  <br/>
COUNTESS OF LOVELACE  <br/>
1815-1852  <br/>
Pioneer of Computing  <br/>
lived here

which, processed through Calibre, produced a nice readable pdf with blockquotes and linebreaks.

For a while after that, I wrote in org-mode and viewed the files within emacs, so I just indented quotations, like so: 

.
    Quandunque i colli fanno più nera ombra,
    Sotto il bel verde la giovane donna
    Gli fa sparir, come pietra sott’ erba.

and then one day I used org-mode’s export-to-html functionality (C-c C-e h o) and I lost all the blockquoting and line-breaks. org-mode needs prose citations surrounded by

#+begin_quote
#+end_quote

to be rendered with <blockquotes> in export-to-html, and if you want it to respect line breaks as well, you need to use

#+begin_verse
#+end_verse

So. Going forward I needed to insert begin/end blocks for new citations, and I had a bunch of older citations I would need to reformat. Here’s what I build with emacs lisp to solve the problem.

The Solution

The Simplest Case

In a new file, about to add a quotation, I want a shortcut to add either #+begin_quote / #+end_quote or #+begin_verse / #+end_verse and put the cursor on the empty line between. This sounds like what yasnippets is designed for, but I wasn’t sure how that would work when I got to converting existing quotations, so I broke out my ~/.emacs.d/mods-org.el file and added two new shortcuts:

(global-set-key (kbd "M-s M-q")
(lambda()
(interactive)
(insert "#+begin_quote")
(newline)
(newline)
(insert "#+end_quote")
(newline)
(previous-line)
(previous-line)))

(global-set-key (kbd "M-s M-v")
(lambda()
(interactive)
(insert "#+begin_verse")
(newline)
(newline)
(insert "#+end_verse")
(newline)
(previous-line)
(previous-line)))

This meets my simplest case requirement and is fairly self-explanatory. Going forward, I can type

M-s M-q ;; q for quote

to get

#+begin_quote
_
#+end_quote

and

M-s M-v ;; v for verse

to get

#+begin_verse
_
#+end_verse

commit

Narrowing the Scope

The fact that I put them in ~/.emacs.d/mods-org.el instead of ~/.emacs.d/key-bindings.el foreshadows the next step: they aren’t global key bindings, I’m only going to use them in org-mode, and in a ruby class they’d just be noise. So, since I’ve already got a hook for entering text-mode (which I’m also using for org-mode), let’s change them from global key bindings to key bindings which are added to org-mode specifically.

I’ve got an ~/.emacs.d/mode-hooks.el file which already has a custom text-mode-hook, so I can add two lines to that:

(defun my-text-mode-hook ()
…
(define-key org-mode-map (kbd "M-s M-q") ‘begin-end-quote)
(define-key org-mode-map (kbd "M-s M-v") ‘begin-end-verse)
)

(add-hook ‘text-mode-hook ‘my-text-mode-hook)

And change the functions, in the ~/.emacs.d/mods-org.el file, from anonymous functions in the global-set-key blocks to the named functions we have just referenced:

(defun begin-end-quote ()
(interactive)
(insert "#+begin_quote")
(newline)
(newline)
(insert "#+end_quote")
(newline)
(previous-line)
(previous-line))

(defun begin-end-verse ()
(interactive)
(insert "#+begin_verse")
(newline)
(newline)
(insert "#+end_verse")
(newline)
(previous-line)
(previous-line))

Now if we’re in org-mode, the key-bindings do what we expect, but if we’re in ruby-mode and type them, or type (C-h k) to get the definition of a key binding and then type (M-s M-v), we get

M-s M-v is undefined

commit

Reformat Existing

That handles the going forward case, but doesn’t handle the case where I’m in an older file and want to reformat an existing quotation. To handle both, I’d like to check when I use the shortcut to see if I’ve got a selected a region or not. If I haven’t, it’s the going forward case, and I should do what I was doing before, but if I have, then presumably I want to put the #+begin and #+end blocks around the selected region.

We can tell this because emacs lisp gives us a function use-region-p which returns true if there is a region selected. The most basic if block in emacs lisp looks like this:

(if (condition)
(do-true-thing)
(do-false-thing))

so in our case we have:

(if (use-region-p)
(begin-end-quote-for-region)
(begin-end-quote-for-new))

and begin-end-quote-for-new is the old begin-end-quote method, and begin-end-quote-for-region looks like this:

(defun begin-end-quote-for-region ()
(interactive)
(insert "#+end_quote")
(newline)
(goto-char (region-beginning))
(insert "#+begin_quote")
(newline))

An extra bit of inwardness here is that the cursor starts at the end of the selected region, so we can just insert “#+end_quote” and it will show up after the end, and (region-beginning) and (region-end) hold the beginning and end of the selected region, so (goto-char (region-beginning)) gets us back to the beginning so we can insert “#+begin_quote” before it.

commit

This gets us to the point where if we’d selected the first quotation and hit M-s M-v, we’d end up with 

#+begin_verse
> ADA  <br/>
COUNTESS OF LOVELACE  <br/>
1815-1852  <br/>
Pioneer of Computing  <br/>
lived here
#+end_verse

which is a definite improvement, but it still has the old formatting codes. Can we get rid of those?

Remove Old Formatting

First off, we only want to do this in the reformat existing case. That’s fine, those two methods (begin-end-quote-for-region and begin-end-verse-for-region) are already separate, so in each of those methods include a (remove-old-formatting) method.

What we want to do in pseudo-code is take the selected region and apply 

s/^> //
s/  <br/>$//

to it. (We only need the second transformation for verse, not quotes, and if these got any more complicated we might want two separate methods, but we can leave them in one for now.)

setq defines a variable.

filter-buffer-substring grabs the text from arg1 to arg2 (and we’re using (region-beginning) and (region-end) which return the start and end of the selected region), and with the optional third argument of t deletes the text after copying it.

After that, we’ve got the contents of the selected region in a variable, “in”, and we can run replace-regexp-in-string on it, taking as arguments search-value, replace-value, and string-to-search in, and using setq to define to variable we’re storing the result in.

Once we’ve made all the changes we need, we use insert the finally modified string back into the buffer.

(defun remove-old-formatting ()
(setq in (filter-buffer-substring (region-beginning) (region-end) t))
(setq out (replace-regexp-in-string "^> " "" in))
(setq out2 (replace-regexp-in-string " <br/>$" "" out))
(insert out2)
)

commit

And at the end of that, we get: 

#+begin_verse
ADA
COUNTESS OF LOVELACE
1815-1852
Pioneer of Computing
lived here
#+end_verse

Which is very nearly there, but not indented. How about as a last step we indent it, so if we’re viewing it in org-mode it still looks like a blockquote?

Indenting

Let’s put this in our remove-old-formatting method, because again it’s something that we’ll only want in the reformat existing case. Given that it’s no longer just removing old formatting, let’s change the method name to fix-old-formatting, and to keep things at the same level of abstraction, let’s put the old remove-old-formatting lines in a new method called remove-old–formatting-code and add a method indent-if-not-indented. So we have:

(defun fix-old-formatting ()
(remove-old-formatting-code)
(indent-if-not-indented)
)

(defun remove-old-formatting-code ()
(setq in (filter-buffer-substring (region-beginning) (region-end) t))
(setq out (replace-regexp-in-string "^> " "" in))
(setq out2 (replace-regexp-in-string " <br/>$" "" out))
(insert out2)
)

Remember that we have some existing citations in the form 

> ADA  <br/>

and some already indented as 

.
    Quandunque i colli fanno più nera ombra,

Further, some of the already indented ones have multiple layers of indentation, and setting a single indentation would break that. So while indent-region itself is simple enough, once again using (region-beginning) and (region-end) to give us the selected region, and the third argument for the number of columns to indent:

(indent-region (region-beginning) (region-end) 4)

we need to make it conditional on it not already being indented, so we end up with:

(defun indent-if-not-indented ()
(setq firstFour (filter-buffer-substring (region-beginning) (+ (region-beginning) 4)))
(if (not (string= firstFour " "))
(indent-region (region-beginning) (region-end) 4)
)
)

using filter-buffer-substring again to grab the first four characters of the region (without the optional third argument so we don’t delete it), and if they aren’t spaces, do the indent. If they are, it’s one of the newer existing quotations and we should leave it as it is, as one of them for instance was pseudo-code 

.
    count = 500 
    day.each do
      wrote_words?(count) ? 
        count += 100 : 
        count -= 100
      count = 100 if count < 100
      count = 1500 if count > 1500
    end

and not doing that check would have clobbered all the internal indenting.

commit

With that, we finally get the desired end result for the older existing quotations: 

#+begin_verse
    ADA
    COUNTESS OF LOVELACE
    1815-1852
    Pioneer of Computing
    lived here
#+end_verse

without clobbering existing indentation for the more recent existing quotations: 

#+begin_verse
    count = 500
    day.each do
      wrote_words?(count) ?
        count += 100 :
        count -= 100
      count = 100 if count < 100
      count = 1500 if count > 1500
    end
#+end_verse

End and Afterthoughts

The current state of my ~/.emacs.d/ is here. It is very much a work in progress. Also, having just looked at Sacha Chua’s more literate emacs config using org-babel, I’m quite tempted to try that out, instead of composing the two or three additional explanatory/introductory blog posts that occurred to me would be helpful as I was writing this up.