Emacs Org-mode Styling, Non-smart Quotes, Zero-width-space, and TeX Input Method

In org-mode we can style inline elements with *bold* (bold), /italic/ (italic), _underlined_ (_underlined_, which won’t show up in html), =verbatim= (verbatim), and ~code~ (code).

But this breaks if the character just inside the styling code is a non-smart single or double quote. =C-c ;= is styled (C-c ;); =C-c '= is not (=C-c '=).

We can fix that by inserting a zero-width space between the apostrophe and the = . The first time, we can put the cursor between the apostrophe and the = and enter C-x 8 RET ZERO WIDTH SPACE RET, at which point =C-c '​= will display as C-c '.

The ZERO WIDTH SPACE part of insert-char (C-x 8) is autocompleting, so we won’t need to type the whole thing, but we’ll probably still want a shortcut if it we find ourselves doing it more than once or twice. Here’s the simplest thing we could do:

(defun my/insert-zero-width-space ()
(interactive)
(insert-char ?\u200B)) ;; code for ZERO WIDTH SPACE
(global-set-key (kbd "C-x 8 s") ‘my/insert-zero-width-space)

After which, if we find ourselves adding inline styles which don’t work because the styled section starts or ends with a quotation mark, we can go to just before or just after the quotation mark, type C-x 8 s, and watch the styles apply as expected.

Thanks, as usual, to Sacha Chua, who explained this in passing. Her .emacs has a more flexible solution which I will probably switch to the moment I need to insert more than one unicode character.

Note that while you can use insert-char to get accented characters (C-x 8 ' e produces é, for instance), and I used to do that, I have since set the input method to TeX (M-x set-input-method RET TeX RET), and now I type \​'e for é, \​th for þ, \​Mu\​epsilon\​nu for Μεν, and so on and so forth.

Automating Boilerplate in Org-mode Journalling

The Problem

My main org-mode files have a header structure like this:

* 2015
** March
*** Friday 6 March 2015
**** 10:00. Emacs coaching with Sacha :emacs:
<2015-03-06 10:00-11:00>
**** 23:37. Emacs progress note
[2015-03-06 23:37]
*** Saturday 7 March 2015
**** 12:55. Stratford King Lear in cinemas :wcss:
<2015-03-07 12:55-15:55>

The two things I do most often are start writing notes in the present, and set up scheduled events in the future. Scheduled events need active timestamps so that they show up in the agenda view:

Week-agenda (W10):
Monday 2 March 2015 W10
Tuesday 3 March 2015
Wednesday 4 March 2015
Thursday 5 March 2015
Friday 6 March 2015
10:00. Emacs Coaching with Sacha :emacs:
Saturday 7 March 2015
12:55. Stratford King Lear in cinemas :wcss:
Sunday 8 March 2015

[How to get the agenda view? Make sure the file you’re in is in org-agenda-files by pressing C-c [, then run org-agenda either by M-x org-agenda or the common (but not out-of-the-box) shortcut C-c a, then press a to get the agenda for the week.]

[Active timestamps are in < >; inactive timestamps are in [ ]. If we want to promote an inactive timestamp to be active and show up in the agenda view, we can run M-x org-toggle-timestamp-type from the inactive timestamp.]

[Start time is duplicated between header and timestamp deliberately: sometimes I review the file in org-mode outline with only the headers. If not for that use-case it would make sense to leave the start time out of the header.]

What I’d like to be able to do is open the file, and then with as little ceremony or setup as possible, start typing.

Adding a New Note

When I’ve already got a date header for the current date, this is trivially simple: I can put my cursor under that date and call M-x my note, having previously defined

(defun my/note ()
(interactive)
(insert "\n\n**** " (format-time-string "%H:%M") ". ")
(save-excursion
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n")))

… and it does pretty much what it says.

The values passed into format-time-string (%Y, %m, %d etc.) will be the values for the current date and time.

save-excursion means “do what’s inside here and then return the point (cursor) to where it was before the block was called”, so it fills in the timestamp and then returns the cursor to the end of the header line, ready for the title to be filled in.

Since after that I’d need to move the cursor back down to below the timestamp, I can be even lazier and prompt for the title and the tags (if any) and fill them in and move the cursor down to where the text should start:

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

If we were just setting one argument, say title, this would be simpler:

(defun my/note-2-eg (title)
(interactive "MTitle? ")

But since we want two, we need to tell interactive it’s getting a list, and then the arguments are set in order.

(defun my/note-2 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))

Just as we can create an inactive timestamp by inserting a formatted date with [ ], we can add tags by inserting a string surrounded by : :. We don’t want to do this if the tag argument was left blank, of course, so we surround it with an unless:

(unless (string= tags "")
(insert " :" tags ":")
)

[Side-note: if you’re doing anything more complicated than this with strings, you probably want Magnar Sveen’s s.el (string manipulation library), which here would let you do (unless (s-blank? tags)… instead.]

At the end of which, because we took out the save-excursion, the cursor is below the timestamp and I’m ready to type.

All good. What if we don’t already have a date header for the current date? Can we auto-generate the date header?

Auto-generating Date Headers

Org-mode has a function called org-datetree-find-date-create. If you pass it in a list of numbers (the month, the day, and the year), and if your header structure is

* 2015
** 2015-03 March
*** 2015-03-06 Friday

then if you called that function passing in (3 7 2015), it would automatically add:

*** 2015-03-07 Saturday

For that matter, if you called it passing in (4 1 2015), it would automatically add

** 2015-04 April
*** 2015-04-01 Wednesday

If you call it passing in a date which is already there, it moves the cursor to that date. So, we could change the format of the org file headers, update our new note function to call the function, and be done.

(defun my/note-3 (title tags)
(interactive (list
(read-from-minibuffer "Title? ")
(read-from-minibuffer "Tags? ")))
(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)
(insert "\n\n**** " (format-time-string "%H:%M") ". " title)
(unless (string= tags "")
(insert " :" tags ":")
)
(insert "\n" (format-time-string "[%Y-%m-%d %H:%M]") "\n\n"))

In the new bit:

(org-datetree-find-date-create (org-date-to-gregorian
(format-time-string "%Y-%m-%d")))
(org-end-of-subtree)

format-time-string “%Y-%m-%d” will return “2015-03-07”, and org-date-to-gregorian will turn that into (3 7 2015), which is the format that org-datetree-find-date-create expects. Determining this involved looking at the source for org-datetree-find-date-create to see what arguments it expected (C-h f org-datetree-find-date-create takes you to a help buffer that links to the source file, in this case org-datetree.el; click on the link to go to the function definition) and a certain amount of trial and error. At one point, before org-date-to-gregorian, I had the also working but rather less clear:

(org-datetree-find-date-create (mapcar ‘string-to-number
(split-string (format-time-string "%m %d %Y"))))

org-end-of-subtree just takes the cursor to the bottom of the section for the date.

And that then works. What about adding new events in the future?

Adding a New Event

org-datetree-find-date-create makes it easier to fill in missing month and date headers to create a new future event:

(defun my/event (date end-time)
(interactive (list
(org-read-date)
(read-from-minibuffer "end time (e.g. 22:00)? ")))
(org-datetree-find-date-create (org-date-to-gregorian date))
(goto-char (line-end-position))
(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")
(save-excursion
(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n")))

There’s a problem I haven’t solved here, which is that org-read-date brings up the date prompt and lets you select a date, including a time or time range. If you select a time, it will be included in the date. If you select a time range, say 19:30-22:30, it ignores the time and the date object returned uses the current time. That’s not what we want.

So when the date prompt comes up:

Date+time [2015-03-07]: _ => <2015-03-07 Sat>

I can put in a new date and start time, say “2015-04-01 11:00”:

Date+time [2015-03-07]: 2015-04-01 11:00_ => <2015-04-01 Wed 11:00-14:00>

and then press enter and that gives me the date and the start time in a single string. We extract the start time and put it into the header like so:

(setq start-time (nth 1 (split-string date)))
(if (string= start-time nil)
(setq start-time ""))
(insert "\n\n**** " start-time ". ")

Where split-string splits the date “2015-04-01 11:00” into a two-element list (“2015-04-01” “11:00”) {n}, and nth 1 returns the second element (list elements, here as elsewhere, are numbered starting from 0), or “11:00” {n}.

If the date had been “2015-04-01” without a start time, line 1 would have set start-time to nil, which would have blown up as an argument to insert (with a “Wrong type argument: char-or-string-p, nil”), so we check for a nil and set it to an empty string instead.

For the agenda view we’ll need the end-time as well, so we grab that as a second argument, and reassemble the date, say “2015-04-01 11:00”, and end-time, say “14:00”, into the active timestamp:

(if (string= end-time "")
(setq timestamp-string date)
(setq timestamp-string (concat date "-" end-time)))
(insert "\n<" timestamp-string ">\n\n"))

though here again, if we had just pressed enter when prompted for end time we would end up with an empty string and an invalid active timestamp, like “<2015-04-01->”, so check whether end-time has been set and build the timestamp string accordingly.

And since this is a future event and we’re probably only filling in the title, we’ll use save-excursion again to fill in the active timestamp and then go back and leave the cursor on the header to fill in the title.

The End

And we’re done. Or we would be, but I would really prefer the header to read “March” instead of “2015-03 March”, and “Saturday 7 March 2015” instead of “2015-03-07 Saturday”. We might need to come back to that.

Emacs, Rvm, and the Important Distinction between Interactive and Non-Interactive Shells

Normally I run the non-graphical version of Emacs in iTerm and switch to a non-Emacs tab to run rspecs and cucumbers at the command line. The other day, since I have feature-mode installed, I typed C-c , v in a feature file to run the cucumber features. The start of the compilation buffer of the results showed:

-*- mode: compilation; default-directory: "~/code/my-project/features/" -*-
Compilation started at Sat Feb 28 00:24:30

rake cucumber CUCUMBER_OPTS="" FEATURE="/Users/seanmiller/code/my-project/features/smoke.feature"
(in /Users/seanmiller/code/my-project)
rake aborted!
undefined method `result' for #<TypeError:0x007fa8ca0f51c0>
/Users/seanmiller/.rvm/gems/ruby-1.9.3-p551@my-project/gems/activerecord-3.2.13/lib/active_record/connection_adapters/postgresql_adapter.rb:1147:in `translate_exception'

with a long stack trace mostly in the activerecord adapters.

The third line (starting “rake cucumber…”) is the command that was run, so we can type M-x eshell to get a shell within Emacs, paste in the line and run it, and there the cucumbers run without error. We can also type M-x compile to get a compile command prompt directly, paste in the line again, and we end up with the error we started with. The compile command prompt and the Emacs shell are behaving differently.

What if we run echo $PATH in both? The shell path starts with several rvm-related directories. The compile prompt path starts with /usr/bin and the rvm-related directories are nearer the end.

That might well do it, but what is causing it? The manual for the compilation shell notes that it brings up a noninteractive shell. The manual for zsh notes that while ~/.zshenv is run for almost every zsh, ~/.zshrc is run only for interactive shells. So if I get the latest version of the rvm dotfiles via rvm get stable --auto-dotfiles and move the two lines that show up in ~/.profile not to my ~/.zshrc file but to my ~/.zshenv file, and then restart Emacs, I can run the cukes from M-x compile or C-c , v and they run cleanly.

What’s more, in the compilation buffer of the test results, the comments referring to the line numbers of feature and step definition files are live links: press enter on them to jump to the lines of the code files. Maybe there is a point to running them in Emacs after all…

Shortcuts to Default Directories

[Now largely historical. See ETA for achieving the same effect with bookmarks, and ETA 2 for achieving the same effect with Hydra, which feels better still.]

Yesterday’s Emacs coaching session with Sacha Chua included a sidebar on jumping to and setting default directories. Sacha’s .emacs.d uses registers for this, and her code sets defaults and org-refile targets.

I found later that I had six shortcuts I’d clean forgotten about, that set the default directory and open either dired or a specific file:

(global-set-key (kbd "C-c C-g C-c")
(lambda ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired ".")))

(global-set-key (kbd "C-c C-g C-h")
(lambda ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org")))

But a forgotten solution doesn’t count, and having to remember triple-decker key bindings doesn’t help either.

A general solution for the second problem is that an incomplete key binding followed by ? opens a help buffer with all the bindings starting with sequence. C-c C-g ? on my machine yielded:

Global Bindings Starting With C-c C-g:
key binding
— ——-

C-c C-g C-c ??
C-c C-g C-d ??
C-c C-g C-e to-english
C-c C-g C-f to-french
C-c C-g C-h ??
C-c C-g C-p ??
C-c C-g C-r ??
C-c C-g C-u ??
C-c C-g C-w ??

We can make this more useful by switching from defining the key binding functions anonymously inline to moving them out to their own named functions, thus:

(defun my/to-emacs ()
(interactive)
(setq default-directory "~/.emacs.d/")
(dired "."))

(defun my/to-today ()
(interactive)
(setq default-directory "~/Dropbox/gesta/")
(find-file "2015.org"))

(global-set-key (kbd "C-c C-g C-c") ‘my/to-emacs)
(global-set-key (kbd "C-c C-g C-h") ‘my/to-today)

Which fixes those two ?? entries:

C-c C-g C-c my/to-emacs
C-c C-g C-h my/to-today

Another niggle is that I’ve overloaded C-c C-g across three different kinds of commands. Let’s make C-x j the shortcut specifically and only for jumping to a new default directory, and extract some duplicate methods while we’re at it. After:

(defun my/to-file (dir file)
(interactive)
(setq default-directory dir)
(find-file file))

(defun my/to-dir (dir)
(interactive)
(setq default-directory dir)
(dired "."))

(defun my/to-gesta-file (file)
(interactive)
(my/to-file "~/Dropbox/gesta/" file))

(defun my/to-emacs-config ()
(interactive)
(my/to-file "~/.emacs.d/" "sean.org"))

(defun my/to-autrui ()
(interactive)
(my/to-dir "~/code/autrui/"))

(defun my/to-gesta ()
(interactive)
(my/to-dir "~/Dropbox/gesta/"))

(defun my/to-today ()
(interactive)
(my/to-gesta-file "2015.org"))

(defun my/to-readings ()
(interactive)
(my/to-gesta-file "readings.org"))

(defun my/to-writings ()
(interactive)
(my/to-gesta-file "writings.org"))

(defun my/to-twc ()
(interactive)
(my/to-dir "~/Dropbox/gesta/twc/"))

(global-set-key (kbd "C-x j e") ‘my/to-emacs-config)
(global-set-key (kbd "C-x j a") ‘my/to-autrui)
(global-set-key (kbd "C-x j g") ‘my/to-gesta)
(global-set-key (kbd "C-x j h") ‘my/to-today)
(global-set-key (kbd "C-x j r") ‘my/to-readings)
(global-set-key (kbd "C-x j w") ‘my/to-writings)
(global-set-key (kbd "C-x j t") ‘my/to-twc)

C-x j ? then yields:

Global Bindings Starting With C-x j:
key binding
— ——-

C-x j a my/to-autrui
C-x j e my/to-emacs-config
C-x j g my/to-gesta
C-x j h my/to-today
C-x j r my/to-readings
C-x j t my/to-twc
C-x j w my/to-writings

Which is an improvement, but the “Bindings Starting With” help menu is not interactive: we still have to either remember the triple-decker key binding, or type C-x j ? to find the list and then type C-x j h (say) to get the specific shortcut we want. It would be nice if we could over-ride C-x j ? and have it prompt us for one more character to select which of the seven jumps we wanted.

Something like this, in fact:

(defun my/pick-destination (pick)
(interactive "ce = ~/.emacs.d/sean.org a = ~/code/autrui/ g = ~/Dropbox/gesta/ h = …/2015.org r = …/readings.org w = …/writings.org t = …/twc/ ? ")
(case pick
(?e (my/to-emacs-config))
(?a (my/to-autrui))
(?g (my/to-gesta))
(?h (my/to-today))
(?r (my/to-readings))
(?w (my/to-writings))
(?t (my/to-twc))))

(global-set-key (kbd "C-x j ?") ‘my/pick-destination)

interactive "c… takes a single character (see further options here), so the conditional is on a character, which in Emacs Lisp is represented by a ? followed by the character {n}. If there were a single conditional we might use (when (char-equal ?e pick) {n}, but since there are seven of them, we look instead for Emacs Lisp’s equivalent of a switch statement, and find it in case (an alias for cl-case).

So what this does, when you type C-x j ?, is provide a prompt of options in the echo area at the bottom of the screen, and if you type one of the significant letters, it uses that shortcut to default.

Purists would probably prefer to bind ’my/pick-destination to something other than C-x j ? (C-x j j say), so that if we ever bound other commands to something starting with C-x j we would still be able to discover them with C-x j ?. It’s also easier to type because it doesn’t need the shift key for the third element. Having demonstrated that we could over-ride C-x j ? if we wanted to, I’m probably going to side with the purists on this one.

And we’re done. Commit.

ETA: Easier Ways To Do It

That works, but since (see Sacha’s comment below) there are always easier ways to do it:

Simpler Code

find-file will either open a file or open dired if it is passed a directory instead of a file, and opening a file or directory will change the default directory. So we can replace all the extracted ’my/to-file and ’my/to-dir methods with simple find-file calls:

(defun my/to-emacs-config ()
(interactive)
(find-file "~/.emacs.d/sean.org"))

(defun my/to-autrui ()
(interactive)
(find-file "~/code/autrui/"))

… etc …

Using Bookmarks (No Custom Code, Even Simpler)

While ’my/pick-destination gets around the non-interactive nature of ? (minibuffer-completion-help, e.g. C-c C-g ? for list of completions to C-c C-g), using straight-up bookmarks lets you do that without custom code:

C-x r m {name} RET ;; bookmark-set ;; sets a {named} bookmark at current location
C-x r b {name} RET ;; bookmark-jump ;; jump to {named} bookmark
C-x r l ;; list-bookmarks ;; list all bookmarks

So having set bookmarks in the same seven places, either files or directories, we could list-bookmarks and get

% Bookmark File
autrui ~/code/autrui/
emacs ~/.emacs.d/sean.org
gesta ~/Dropbox/gesta/
now ~/Dropbox/gesta/2015.org
readings ~/Dropbox/gesta/readings.org
twc ~/Dropbox/gesta/twc/
writings ~/Dropbox/gesta/writings.org

And this list is interactive. Or we could call C-x r b (bookmark-jump) and start typing the destination we want. (If we get bored of typing, we can reduce the bookmark names to single characters, remembering that C-x r l (list-bookmarks) will give us the translation table if we forget, and then we’re back to being five keystrokes away from any bookmark, without having to add any extra code.)

Using Bookmark+ (And Describing and Customizing Faces)

If we want to be able to do more advanced things with bookmarks – tag them, annotate them, rename them, run dired-like commands on the bookmark list – we can grab the Bookmark+ package and (require ’bookmark+) in our .emacs. (Having done that, if we press e by a line in the bookmark list, for instance, we can edit the lisp record for the bookmark, to rename it or change the destination or see how many times it has been used.)

One problem I had with bookmark+ is that the bookmark list was displaying illegibly in dark blue on a black background. To fix this, I needed to move the cursor over one of the dark blue on black bookmark names and type M-x describe-face, and it reported the face (Emacs-speak for style {n}) of the character, in this case Describe face (default `bmkp-local-file-without-region'):. Pressing enter took me to a buffer which described the face and provided a customize this face link at the end of the first line. I followed (pressed enter) that link to get to a customize face buffer which let me change the styling for that element. On the line:

[X] Foreground: blue [ Choose ] (sample)

I followed (pressed enter on) Choose, it opened a buffer of colour options, I scrolled up to a more visible one, pressed enter again, and got back to the customize face buffer, then went up to “Apply and Save” link and pressed enter again there. Going back to the bookmark list, the bookmarks were visible. The change is preserved for future sessions in the init.el file:

(custom-set-faces
;; custom-set-faces was added by Custom.
;; If you edit it by hand, you could mess it up, so be careful.
;; Your init file should contain only one such instance.
;; If there is more than one, they won’t work right.
‘(bmkp-local-file-without-region ((t (:foreground "green")))))

ETA 2: Using Hydra Instead

Heikki Lehvaslaiho suggested using hydra instead, and as the bookmark solution required a couple of extra keystrokes and I’d been curious about hydra anyway, I thought I’d give it a go.

(require ‘hydra)
(global-set-key
(kbd "C-c j")
(defhydra hydra-jump (:color blue)
"jump"
("e" (find-file "~/.emacs.d/sean.org") ".emacs.d")
("c" (find-file "~/.emacs.d/Cask") "Cask")

("a" (find-file "~/code/autrui/") "autrui")
("h" (find-file "~/Dropbox/gesta/2015.org") "hodie")
("r" (find-file "~/Dropbox/gesta/readings.org") "readings")
("w" (find-file "~/Dropbox/gesta/writings.org") "writings")
("t" (find-file "~/Dropbox/gesta/twc/") "twc")))

I like it. We’re back down to three keystrokes from five and, since we’re providing labels for the commands, we also get the descriptive and interactive and self-updating index which was the big advantage of the bookmark route. If you press C-c j and don’t immediately carry on, it prompts:

jump: [e]: .emacs.d, [c]: Cask, [a]: autrui, [h]: hodie, [r]: readings, [w]: writings, [t]: twc.

Much better. Thanks for the prompt, Heikki!

Literate Emacs Configuration

What?

Literate programming, proposed by Donald Knuth in 1984 {n}, suggests that

Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Literate Emacs configuration? Lets us build our .emacs.d files in .org files instead of .el files, so that with very little extra effort they can look like this.

In case that doesn’t speak for itself:

  • with org-mode’s headers and subheaders, you can quickly find your way around a large file, hiding all but the subheader content you care about
  • with org-mode’s links, the links to the snippets from What the .emacs.d!? or Avdi’s Emacs Reboot are live links
  • with org-mode’s export to html, you can weave an even more readable version of the config.
  • since it’s org-mode, you can put TODO statuses or tags on headers as well, so you can easily record the status of your practice on some shortcuts, or tag parts of the config as :experimental:.

How?

The simplest version really does work out of the box.

In your init.el file, include an org-babel-load-file reference to the org file you’re going to use, in my case ~/.emacs.d/sean.org:

(org-babel-load-file "~/.emacs.d/sean.org")

Then, create the org file, and put whatever bits of config that you want in emacs-lisp source blocks, like so:

#+BEGIN_SRC emacs-lisp
  ... code here ...
#+END_SRC

What this will do is export the source code using org-babel-tangle (more on that later) into a file ~/.emacs.d/sean.el, and it will then load the resulting file using load-file. It is effectively the same as

(load-file "~/.emacs.d/sean.el")

except that it checks the timestamps of the sean.el and sean.org files and if the sean.org file was changed later, it re-extracts the sean.el file.

(Since sean.el is a generated file you will probably want to add it to your project’s .gitignore so it isn’t committed.)

And in the sean.org file you can put headings and subheadings to make it easy to navigate: in my first try, because my Emacs config had previously been across several files, I used the file names as subheadings, ending up with something like this:

* Emacs Config
** Defaults…
** Mode hooks…
** Global key bindings…
** Project-specific shortcuts…
** Zenburn theme…
** Emacs server and Emacsclient…
…etc

Check the Html Output

It is the case that if you export that from org to html right now (C-c C-e h o / org-export-dispatch) the h1 title will be the filename, which is unhelpfully just “sean”. Give it a better name with an explicit title export setting at the top of the file:

#+TITLE: Sean Miller’s Emacs Configuration

Re-export, and the html file has that title, and, by default, a linked table of contents derived from the headers and subheaders. (Other export options are described here.)

Run the Tests: A Catch

The next thing that I did was try to run the tests (tests introduced here, test runner introduced here), after changing the initial load-file line in the test to point to the generated sean.el file:

(load-file "sean.el")

And… I got an abnormal exit from ert-runner because the server-start from another part of my Emacs config prevented ert-runner from starting a server or running any of the tests.

We can fix that by breaking apart the emacs-lisp blocks from sean.org into two separate files, one of settings and one of code-under-test, and then in the test we can load the code-under-test file on its own.

To do this, we add a :tangle argument to the #+BEGIN_SRC emacs-lisp blocks in the sean.org file, with a filename argument. Code that is not under test gets the begin line:

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-settings.el

And code under test gets the begin line

#+BEGIN_SRC emacs-lisp :tangle ~/.emacs.d/tangled-code.el

Then, when we make changes to the sean.org file, on save we can run the org-babel-tangle command and it will export the emacs-lisp blocks into ~/.emacs.d/tangled-settings.el and ~/.emacs.d/tangled-code.el respectively.

We can then remove the org-babel-load-file line from init.el and replace it with:

(load-file "~/.emacs.d/tangled-settings.el")
(load-file "~/.emacs.d/tangled-code.el")

So that the next time we restart Emacs it will pick up the two built files, and if we change the load in our test file to be

(load-file "tangled-code.el")

And run our tests again, this time they run fine, so we have literate Emacs config and running tests.

(You’ll want to change your project’s .gitignore again to include the new “tangled-” files.)

Why “tangled”? It comes from the terminology of literate programming: the literate source file can be “tangled” to produce machine-readable code (exporting the source code blocks into .el files), and “woven” to produce formatted documentation. Knuth later admitted {n} that the echo of Walter Scott’s “Oh, what a tangled web we weave when first we practise to deceive” was entirely deliberate.

Automating the Manual Step We Just Introduced

When we fixed the catch we lost the automatic updating of the .el files when we changed the .org file. Now we need to remember to run org-babel-tangle manually after changing the .org file.

Manual steps are invariably forgotten, so let’s automate. The simplest hook would be:

(add-hook ‘after-save-hook ‘org-babel-tangle)

But we only want to tangle if we’ve just saved the Emacs config org file, so let’s build a new function to check that before running tangle, and hook to that instead:

(defun my/tangle-on-save-emacs-config-org-file()
(when (string= buffer-file-name (file-truename "~/.emacs.d/sean.org"))
(org-babel-tangle)))

(add-hook ‘after-save-hook ‘my/tangle-on-save-emacs-config-org-file)

buffer-file-name returns the absolute path of the name of the file the buffer is visiting. file-truename converts ~/.emacs.d/sean.org into an absolute path so that the string comparison works.

And with that, we’re done. Commit.

Emacs Org-mode: More on Todos and Clock Reports

Previously, on The Wandering Coder, we used org-mode to build a more automatable answer to the questions “what did I work on last Friday?” and “what sorts of tasks did I work on last week?”. At the end of that entry, we had a simple file that let us clock time against tasks, set the TODO status of those tasks, and produce clock table reports of the result.

Here are three additional points that were not on that critical path but are still nice-to-haves:

Customizing TODO Keywords

TODO and DONE are the default TODO keywords, but you can over-ride that by creating a line #+TODO at the top of the file with your own list of custom todo keywords, for instance:

#+TODO: TODO STARTED BLOCKED QA DONE

And then, after pressing C-c C-c to reload and use the new values, C-c C-t will run through not the default of TODO and DONE but the new list of custom keywords.

Annotating TODO State Transitions

If you have a custom #+TODO line, you can add a (!) or an (@) to annotate state transitions, for instance,

#+TODO: TODO STARTED(!) DONE(@)

The (!) indicates that when a task is toggled into STARTED, the code automatically records a timestamp at the time of the state change.

The (@) indicates that not only does it record a timestamp when a task is toggled into DONE, it prompts the user for a note about the state change.

So if we had had that todo list in place for

** back-end task #121

and toggled it through TODO to STARTED to DONE, and added a note for the DONE state, we would end up with:

** DONE back-end task #121
– State "DONE" from "STARTED" [2015-02-06 Fri 15:00] \\
Note about task being done.
– State "STARTED" from "TODO" [2015-02-06 Fri 9:00]
CLOCK: [2015-02-06 Fri 9:00]–[2015-02-06 Fri 12:00] => 3:00
CLOCK: [2015-02-06 Fri 13:00]–[2015-02-06 Fri 15:00] => 2:00

More org-clock-table customizations

Exploring the clock table documentation reveals many more options to add to the #+BEGIN: line to customize the results than just :block today to see what was done today, or :block today-1 to see yesterday, or :tags “unplanned” to see the tasks tagged as :unplanned:.

If for instance we want the report for last week but broken down day-by-day, for instance, we could add to the #+BEGIN: line (remember also to press C-c C-c to update the results):

:block lastweek :step day

and get something like:

Daily report: [2015-02-09 Mon]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |

Daily report: [2015-02-10 Tue]
| Headline | Time | |
|————————–+——–+——|
| *Total time* | *7:00* | |
|————————–+——–+——|
| Work | 7:00 | |
| \emsp back-end task #125 | | 7:00 |

Daily report: [2015-02-11 Wed]
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *5:00* | |
|—————————+——–+——|
| Work | 5:00 | |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |

etc…

If your work sprint runs from Wednesday to Wednesday instead of Monday to Friday, as mine did last year, you can add :wstart 3 (instead of 1, which is Monday and the default), and then the weekly report from

:wstart 3 :block lastweek :step day

would begin instead with

Daily report: [2015-02-11 Wed]

Emacs Documentation: Info-mode, Describe-mode

Links to Emacs Lisp or Org-mode documentation in the blog are to the online versions at https://www.gnu.org/software/emacs/manual/html_node/elisp/ and http://orgmode.org/manual/, but you can also read the manuals within Emacs.

C-h i opens an info-mode buffer with a list of available manuals, and if you scroll down to Elisp and press enter you can read the Emacs Lisp Reference Manual, or to Org Mode and press enter you can read the Org Mode Manual.

To determine what key shortcuts are available in a buffer, say because you’re in an info-mode buffer deep in the Org Mode Manual and you’re trying to get back up to the list of manuals, press C-h m to open another buffer in describe-mode, which will explain that from info-mode a ^ goes up a from a node to its parent node, and, when you’re finished, q quits and restores the info-mode buffer to whatever was in it before you started reading the manuals.

C-h m results are customized to the type of buffer and the modes active within it: from the org-mode buffer I’m typing this in, for instance, they include:

C-c M-p d org2blog/wp-post-buffer
C-c M-p p org2blog/wp-post-buffer-and-publish
etc…

M-s M-e my/begin-end-src-emacs-lisp
M-s M-q my/begin-end-quote
etc…

Emacs Org-mode: Tasks, Todos and Timings

The Problem

You’re in a Monday morning standup meeting. Everybody’s update begins “Well, I don’t really remember what I did on Friday, but…”

The Simplest Solution

In a notebook or a text file, write it down. Add a line for each task or ticket you worked on in a day.

– Friday 6 February
– back-end task #122
– production bug #123
– front-end task #124

This solves the Monday-morning-I-don’t-remember-Friday problem, but what if you’re trying to get better at estimating and you want to know how long each task took? What if you want a report on how much of it was planned tasks versus unplanned interruptions? And how do you automate as much of this as possible?

Enter Org-mode

From the first line of the manual:

Org is a mode for keeping notes, maintaining TODO lists, and doing project planning with a fast and effective plain-text system.

Create a file called work.org. (The .org suffix will automatically put it in org-mode when reading it in Emacs.)

* Work
** back-end task #121
** back-end task #122
** production bug #123
** front-end task #124

Outline

The file is an outline: lines starting with asterisks are headers, and the more asterisks, the deeper the sub-header, so in html terms (or explicitly if you export an org-mode file to html):

* h1 header ;; => <h1>h1 header<//h1>
** h2 sub-header ;; => <h2>h2 sub-header</h2>

Any line not starting with an asterisk is content contained within the header above it, so you can start adding notes about production bug #123:

* Work
** back-end task #121
** back-end task #122
** production bug #123

Affecting servers X, Y
Tried A, B, C

** front-end-task #124

To hide anything under a header (notes or any subheaders) you can press tab at the beginning of the line, so to hide the details under “** production bug #123” you would press tab at the beginning of the “** production bug #123” line, which would change the display to:

* Work
** back-end task #121
** back-end task #122
** production bug #123…
** front-end-task #124

Pressing tab again at the beginning of the “** production bug #123…” line will re-reveal the hidden content.

You can also press shift-tab anywhere to cycle between displaying just the top-level header, all the headers and subheaders, and all the headers, subheaders, and text.

Links

If your work tasks are tickets in JIRA or some such and you want links direct from the org-mode file, you can:

  • use the URL of the link (e.g. http://devnull.atlassian.net/task-121) as the text, instead of “back-end task #121”. A URL in an org-mode document will show up as a link and pressing C-c C–o (org-open-at-point) will open a web browser to that URL.
  • use C-c C-l (org-insert-link) to build a link: it will ask you for the link (insert the URL), and then again for description. This will result in, e.g., back-end task #121. C-c C-o will open this link too.

Tracking time against tasks

You arrive Friday morning, look at your list, and start work on back-end task #121. To start tracking the time spent, go to the header back-end tasks #121 and press C-c C-x C-i, which runs org-clock-in, and starts a clock on that header.

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]

When you finish the task or stop working on it, press C-c C-x C-o to run org-clock-out, which will stop the running clock on this task:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:00] => 2:00

Alternately, if you run org-clock-in on another task, it will stop the clock on the previous task before starting it on the new one:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 10:45] => 1:45
** back-end task #122
CLOCK: [2015-02-06 Fri 10:45]

If you leave a task and come back to it (after lunch, say), you can clock out when you leave and clock back in again when you come back, and it starts a new clock:

* Work
** back-end task #121
CLOCK: [2015-02-06 Fri 12:45]
CLOCK: [2015-02-06 Fri 09:00]–[2015-02-06 Fri 11:45] => 2:45

When multiple clock records become distracting, remember that pressing tab at the beginning of a header hides everything underneath that header, so the last example can be reduced to:

* Work
** back-end task #121…

Reporting

You’ve built up your list for a week, adding new tasks as you go and clocking in and out of them to keep track of time spent. You come in on Monday the 16th and it’s time for morning standup. To get from the list of tasks a report of what you worked on on Friday the 13th, you have two options: use org-clock-report or org-agenda. We’ll discuss org-agenda in a later post.

C-c C-x C-r on a header runs org-clock-report on all the content below that header, so running it here on the * Work header would autogenerate a report of all the times for all the tasks, which over a week might look like:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55]
| Headline | Time | |
|—————————+————+——|
| *Total time* | *1d 16:00* | |
|—————————+————+——|
| Work | 1d 16:00 | |
| \emsp back-end task #121 | | 5:00 |
| \emsp back-end task #122 | | 2:00 |
| \emsp production bug #123 | | 2:00 |
| \emsp front-end task #124 | | 5:00 |
| \emsp back-end task #125 | | 7:00 |
| \emsp production bug #126 | | 2:00 |
| \emsp back-end task #127 | | 3:00 |
| \emsp hotfix #128 | | 1:00 |
| \emsp front-end task #129 | | 8:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

In this case, that’s all the tasks for eight days, not all the tasks just for last Friday. To limit it, we can modify the auto-generated #+BEGIN line by adding :block today-3 at the end. Press C-c C-c on that line to update the table, and we get instead:

* Work
#+BEGIN: clocktable :maxlevel 2 :scope subtree :block today-3
#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|—————————+——–+——|
| *Total time* | *7:00* | |
|—————————+——–+——|
| Work | 7:00 | |
| \emsp front-end task #129 | | 2:00 |
| \emsp front-end task #130 | | 5:00 |
#+END:

** back-end task #121

And there’s our standup report. Well, almost. It would be nice if it also told us whether each task was done or not.

Status of tasks

As part of org-mode’s handling of TODO lists, any header can be turned into a TODO item. With the cursor on a header line, press C-c C-t (org-todo) three times. It will cycle from (say)

** back-end task #121

to

** TODO back-end task #121

to

** DONE back-end task #121

and back to

** back-end task #121

And the TODO and DONE fields show up in the clock reports, so if these TODO keywords were set and front-end task #129 were DONE and front-end task #130 were TODO, the report would look instead like

#+CAPTION: Clock summary at [2015-02-16 Mon 8:55], for Friday, February 13, 2015.
| Headline | Time | |
|——————————–+——–+——|
| *Total time* | *7:00* | |
|——————————–+——–+——|
| Work | 7:00 | |
| \emsp DONE front-end task #129 | | 2:00 |
| \emsp TODO front-end task #130 | | 5:00 |

And there’s our standup report.

Planned versus Unplanned: Tagging Tasks

The one other thing in our initial problem was to be able to report on planned tasks versus unplanned interruptions, perhaps as part of figuring out velocity or how much unexpected work or emergencies disrupted the work of the sprint. To handle this we can add tags to headers, and use those as filters for the org-clock-report.

Suppose

** production bug #123
** production bug #126
** hotfix #128

are unplanned interruptions. With the cursor on those headers, press C-c C-q to add a tag. It will prompt for a tag name: you can just type in unplanned in each case or set up a list of possible tags like so

#+TAGS: unplanned

and then (after reloading with C-c C-c), you get autocompletion on tag names. Having added the tags, the collapsed table will look like:

** back-end task #121…
** back-end task #122…
** production bug #123 :unplanned:…
** front-end task #124…
** back-end task #125…
** production bug #126 :unplanned:…
** back-end task #127…
** hotfix #128 :unplanned:…
** DONE front-end task #129…
** TODO front-end task #130…

At that point, you can use org-clock-mode using :tags on the #+BEGIN: line and the tag match syntax, so you can retrieve unplanned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|—————————+——–+——| |
| *Total time* | *5:00* | | |
|—————————+——–+——| |
| Work | 5:00 | | |
| \emsp production bug #123 | | 2:00 | |
| \emsp production bug #126 | | 2:00 | |
| \emsp hotfix #128 | | 1:00 | |
#+END:

and planned tasks like so:

#+BEGIN: clocktable :maxlevel 2 :scope subtree :tags "-unplanned" |
#+CAPTION: Clock summary at [2015-02-15 Sun 19:22] |
| Headline | Time | | |
|——————————–+————+——| |
| *Total time* | *1d 11:00* | | |
|——————————–+————+——| |
| Work | 1d 11:00 | | |
| \emsp back-end task #121 | | 5:00 | |
| \emsp back-end task #122 | | 2:00 | |
| \emsp front-end task #124 | | 5:00 | |
| \emsp back-end task #125 | | 7:00 | |
| \emsp back-end task #127 | | 3:00 | |
| \emsp DONE front-end task #129 | | 8:00 | |
| \emsp TODO front-end task #130 | | 5:00 | |
#+END: |

And there, in addition to our standup report, is report on planned tasks vs. unplanned interruptions. The entire initial problem is now solved.

Refactoring “Beginning Emacs Lisp”: II

Now that we have test coverage and a test runner, we can refactor the begin/end code.

First off, let’s namespace it. Instead of defun begin-end-quote (), let’s make it defun my/begin-end-quote ().

Why? There is another library with a function called begin-end-quote: I’m not using it, but if I were, after namespacing I wouldn’t have to worry about anything breaking because of the name collision.

The coding conventions suggest namespacing packages for widespread use with -, not / (q.v. typopunct-mode ’s typopunct-insert-quotation-mark), but namespacing “less formal” code with / seems a useful distinction. Sacha Chua suggested a further convention for Emacs config files of my/ instead of (say) sean/, since .emacs.d code is so often copied and borrowed and it would make it look more consistent after mixing code from multiple sources.

Run the tests: they still pass. Hurrah.

Almost all of the code between begin-end-verse and begin-end-quote is the same. I made them separate to begin with because I thought they would be more different, but they aren’t. So we can make everything but the outermost function take an argument of “quote” or “verse” and remove the duplication. (This will also let me quickly add variants like #+begin_src emacs-lisp and #+begin_example, which I’ve already needed while writing this blog.)

Again, run the tests, they still pass. Commit with this change.

Further quick points: newline does insert a carriage return, but insert “\n” would do the same thing, and feels more intuitive. Likewise, while previous-line and next-line do what they say, the more usual way would be to run forward-line with a positive or negative integer argument.

There is also duplication between my/begin-end-selected-region and my/begin-end-no-selected-region: they both print the #+begin and #+end tags, and if there’s a selected region it is reformatted, and if there isn’t, after printing the tags it moves the cursor back up between the tags. We can collapse the two methods into a single method with two conditionals.

(defun my/begin-end (variant)
(interactive)
(let ((cited-string "\n"))
(when (use-region-p)
(setq cited-string
(my/remove-old-citation-formatting (buffer-substring-no-properties (region-beginning) (region-end))))
(delete-region (region-beginning) (region-end)))
(insert "#+begin_" variant "\n"
cited-string
"#+end_" variant "\n"))
(unless (use-region-p)
(forward-line -2)))

The simpler conditional is at lines 11-12: (use-region-p) returns true if there is a region selected, and if there isn’t, it moves the cursor up two lines.

The one in lines 3-10 is more complicated.

In lines 3-7, we’re using a let statement to set cited-string initially to \n, which is what it should be if there’s no selection, and then we’re checking to see if there is a selected region: if there is, we’re resetting cited-string to the value of the selected region with the old formatting code stripped out, and then we’re deleting the selected region.

At the beginning of line 8, then, we’ve deleted the selected region if there was one, and until the end of the body of the let at the end of line 10, cited-string contains either a return character (if there was no selection) or the reformatted contents of the selected region (if there was a selection). Because of that, we can insert it between the tags either way.

As part of this, we also modified the my/fix-old-formatting method from one that dealt with the selected region in place to one which took a string and returned a modified string. All the selected region manipulation now takes place in the my/begin-end function.

Re-run the tests: they still pass. Excellent. Commit.

At this point, the refactoring is done, but it is now trivially easy to add new tests and implementations and key-bindings for #+begin_example / #+end_example and #+begin_src emacs-lisp / #+end_src.

The test for #+begin_src emacs-lisp breaks because for _src, the beginning tag and the end tag are different. The simplest way to get this passing is to pass in two arguments to my/begin-end, which for all but src will be identical, like so:

(defun my/begin-end-example ()
(interactive)
(my/begin-end "example" "example"))

(defun my/begin-end-src-emacs-lisp ()
(interactive)
(my/begin-end "src emacs-lisp" "src"))

(defun my/begin-end (begin-tag end-tag)

(insert "#+begin_" begin-tag "\n"
cited-string
"#+end_" end-tag "\n"))

At this point, the code does everything we need, the tests pass, and we drove out the new functionality we needed with tests. We’re done. Final commit.

Emacs Lisp: Adding Tests: ert-runner and overseer

In the previous Adding Tests post I found that running M-x ert on its own did not always pick up added or removed tests, but that a test running tool called ert-runner might fix this. It does, and adding another tool called overseer makes it easy to run tests or subsets of tests without leaving Emacs.

Here’s how I got there:

  1. Install Cask to manage project dependencies, as ert-runner uses it to run the tests from the command-line. I found that the default brew install cask didn’t set up ~/.cask/cask.el properly, but the longer curl -fsSL https://raw.githubusercontent.com/cask/cask/master/go | python did.
    • After that, the usage instructions mostly worked out of the tin.
      • Calling cask init --dev sets up a Cask file (think Gemfile from ruby projects) with a block of development dependencies. We aren’t using ecukes or el-mock yet, and we’ll add overseer.
      • I also took out the (package-file "TODO") line, as it stopped cask install from running.
      • I had borrowed package-loading from bbatsov’s excellent Prelude: for now I’ve commented that out and put my Emacs package dependencies into the Cask file.
      • Running cask install puts all the packages in a .cask directory, so you might want to add that to .gitignore.
      • commit for adding Cask
  2. With ert-runner installed by Cask, from the command line, run cask exec ert-runner init. This creates a test directory and an empty test/test-helper.el file. Previously, our tests for the begin/end functionality were in the same file as the code: for ert-runner to be able to find them, they need to be in the test directory, in a file ending -test.el. So we move the tests from mods-org.el to test/mods-org-test.el.
    • The newbie / coming from Ruby mistake that I made at this point was to go to the command line and run cask exec ert-runner, expecting it to pick up the files automatically. This of course fails. We need to put a (provide 'mods-org) at the bottom of the mods-org.el file, and a (load-file "mods-org.el") at the top of test/mods-org-test.el.
    • Having done that, we can run the tests, and they all pass. Further, if some of them fail, we can do the usual thing: comment out all but one of them and re-run and, unlike with M-x ert, it picks up the changes and just runs a single test.
    • commit for reorganizing tests
  3. When going out to the command line or popping open a shell to run cask exec ert-runner every time proves irksome, we can take advantage of the fact that we also added overseer, which gives us some handy shortcuts. With a single test file so far, I have been using C-c , b to run all the tests from the buffer.

Thanks to Sacha Chua for unblocking me at one point, and sharing a draft post which takes things further into continuous integration and test coverage tools.