Automatic Github Pages generation from Sphinx documentation

Sphinx is a very common documentation tool which gobbles up ReStructuredText and other free-form markup formats and outputs great HTML, PDF and other formats. It is meant for reference manuals and API documentation due to its good integration with source code (especially Python). The libuv book is written using Sphinx so that it looks so good with minimum effort.

Sphinx uses make to generate the HTML, which is great. The only problem is that deploying this to Github Pages requires multiple commands to switch branches to gh-pages, pull in the source text, then cleanup the working copy and switch back to master. This is boring after about one time, so I automated it, and I think other projects can benefit from it as well. Once you follow the instructions, running:

make gh-pages

will take the latest commit, switch to the gh-pages branch, generate HTML, push it to Github, then clean everything up and switch back to master.

NOTE: You need to commit or revert any working copy modifications before running this.

One-time commands

These steps only need to be run the first time when you want to generate Github Pages. First setup the branch to have no parents. Let me stress again the importance of making sure all changes are committed! Otherwise they’ll be lost.

$ cd repo
$ git checkout --orphan gh-pages
$ git rm -rf .
$ echo "First commit" > index.html
$ git add .
$ git commit -m "Just to create the branch."
$ git push origin gh-pages

Now the gh-pages branch is setup. We can start generating the actual pages instead of the current index.html.

Set the source files

Edit the Sphinx Makefile. Add a variable GH_PAGES_SOURCES. This should be a list of the files/directories that contain the documentation sources. This will usually be only source which contains the Sphinx reST docs, but if you are embedding external code or images, those directories have to be listed as well. In addition the Makefile has to be in the list. For the libuv book it is:

GH_PAGES_SOURCES = source code libuv Makefile

Add the target

Create a target gh-pages with the following commands (remember to use TABs in Makefiles):

gh-pages:
    git checkout gh-pages
    rm -rf build _sources _static
    git checkout master $(GH_PAGES_SOURCES)
    git reset HEAD
    make html
    mv -fv build/html/* ./
    rm -rf $(GH_PAGES_SOURCES) build
    git add -A
    git ci -m "Generated gh-pages for `git log master -1 --pretty=short --abbrev-commit`" && git push origin gh-pages ; git checkout master

Here is how it goes. The checkout simply switches branches. Then we remove all the old data to prevent any rebuilding artifacts. Since the gh-pages branch won’t have any of your original data, but only the HTML output, we need to pull the sources from the master branch. Then we generate the HTML. We move these from the build folder to the top level. Then we remove all the sources and the now empty build folder. We stage all the changes. Finally the last line generates a commit message for the gh-pages changes which is the first line of the latest commit on master and pushes to Github. The reason the git checkout master command is on the same line and semi-colon separated; if the push were to fail for some reason (network error, DAG inconsistency), I want to be returned to master in my working copy. If you don’t want this to happen, feel free to move it to a new line on its own.

Done!

You can now get back to your main task - writing great documentation. Whenever you have an urge to show it to the world, simply run make gh-pages and your latest documentation is served fresh!

Playing with Go

With Go 1 being released, I’ve been playing with the language once again. As a long weekend hack, I created a clone of the literate programming Docco tool in Go – quite naturally called Gocco.

This blog post chronicles my feelings about the language, and some rough spots I got stuck in.

I started out with a direct translation of the original CoffeeScript to Go. Go is remarkably unlike C and more like high level languages. Much of the translation was automatic and is not very different from the CoffeeScript source, except the pervasive use of []byte rather than string.

What I loved about Go was the ability to directly import and install packages from various hosting services like GitHub, Google Code and Bitbucket. This combined with the go command makes package management a breeze. What I would really like though is for Go to switch to default local-to-global search path semantics, like npm. You can do it using GOPATH, but making it the default would be better. I am also not sure how you can specify which version of your dependencies to install.

The inbuilt templating support is also a great tool, I didn’t have to hunt around for a templating library, and the syntax was very clean and similar to Jinja2 or Mustache. There seems to be some bug in the ParseFiles method though, as it kept complaining that the template was incomplete or empty, but when I manually read the same file into a string and called Parse, it worked.

The tooling support is excellent, with gofmt and go ensuring everything is ‘standardised’ and makes others code much easier to read.

Coming to the slightly wonky/undefined/undocumented/(may actually be my fault) parts…

I still haven’t really figured out Go’s package system. The go/build package says it allows introspection over packages, like figuring out the installation path (Similar to Python’s __file__). But I couldn’t get this to work with my code, especially when it was not installed. I wanted it so I could keep the HTML template and CSS in separate files, and use the package path to figure out where they were kept. I finally got very frustrated and just stuck them into a go file as multi-line strings.

Syntactically, multiple return values are a good way to signal errors without exception handling, there really should be a way to ignore the second, error, parameter. Not being able to do that, makes composing functions impossible. Similarly the unused variables and imports errors can get annoying while developing, when there is lots of stub code. There is a workaround but still! :)

Still, it was a lot of fun using Go. I originally wanted to implement Docco in C, but the lacking standard library and types support was scaring me. Now I will reach for Go when it comes to systems programming.

A thanks to Russ Ross, for an excellent Markdown implementation.