Testing Vim Plugins on Travis CI with RSpec and Vimrunner

I have been using Vim as my primary text editor for several years but it was only recently that I decided to try and write my own Vim plugin. My aim was simple: to create a function that would run the tests for whatever file I was working on. If the file was a test itself then simply run it but if it wasn’t, try to find the corresponding test and run that.

Having heard Vim script (sometimes referred to as VimL) is a rather esoteric language, I pored through the official Vim manual section and Steve Losh’s “Learn Vimscript the Hard Way” and got stuck in.

The code was simple at first but, as it progressed, I found myself programming and then firing up Vim to check whether everything worked as expected. At the recent London Ruby User Group “TDD Fishbowl” debate, this method of verifying code correctness was derisively referred to as “Refresh-Driven Development”. Mindful of this bad practice and having grown tired of manually inspecting functions in Vim, I decided to look into solutions for not only automating the testing of Vim script but also techniques for test driving development.

My initial searches resulted in a Stack Overflow post about Vim script unit test frameworks but none of the solutions appeared particularly popular or actively maintained. However, by adding “driving” to my search terms, I came across Andrew Radev’s “Driving Vim with Ruby and Cucumber”.

I recommend you read Andrew’s full post but the summary is that he wrote a gem called Vimrunner which uses Vim’s client-server functionality to drive Vim from Ruby. Using this gem, he was able to create an extension to Cucumber to write acceptance tests for his Vim plugins.

This was almost exactly what I was looking for except that I prefer to use RSpec for small libraries rather than Cucumber. Luckily, the core of cucumber-vimscript is very simple, so I decided to explore using only Vimrunner and RSpec to test-drive my plugin.

To begin with, I had the following directory layout with the standard files:

.
├── plugin
│   └── runspec.vim
├── autoload
│   └── runspec.vim
└── doc
    └── runspec.txt

In order to get up and running with these Ruby testing frameworks, I added a Gemfile with the following contents:

source 'https://rubygems.org'
gem 'vimrunner', '0.1.0'
gem 'rspec',     '~> 2.9.0'

The actual process of testing the plugin is as follows:

  1. Start a Vim server by invoking Vim with an explicit servername;
  2. Load our plugin under test into the server (viz. put it in Vim’s runtime path and execute the main script);
  3. Start sending commands to our Vim server using remote-expr and check the results for correctness;
  4. Quit the Vim server.

If we’re writing a spec called spec/runspec.vim_spec.rb then steps 1, 2 and 4 are straightforward thanks to Vimrunner:

require 'vimrunner'

vim = Vimrunner.start
vim.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')

describe "runspec.vim" do
  after(:all) do
    vim.kill
  end
end

As you can see, we require the gem and then start a new Vim server with start. We add our plugin to the server using the add_plugin helper method and pass the directory (given relative to our current test file) and the relative path of the main plugin script. We also add an RSpec after hook to shut down the server once we are done.

With this in place, we can start writing some basic tests.

As the point of my plugin is to run tests by executing a shell command with :!, it could prove difficult to exhaustively test. However, instead of focussing on the actual execution, I can test the parts of the plugin leading up to that final stage. In particular, let’s look at the part of the plugin that tries to find the most appropriate spec for the currently open file.

Before I started writing tests, this function was private and accessible only to the plugin thereby making it difficult if not impossible to test. By switching to using Vim’s autoload functionality, I could now access (and test) the function from outside the plugin but without worrying too much about namespace clashes.

Here’s the first (admittedly simple) spec:

require 'vimrunner'

vim = Vimrunner.start
vim.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')

describe "runspec.vim" do
  after(:all) do
    vim.kill
  end

  describe "#SpecPath" do
    it "returns the current file if it ends in _spec.rb" do
      vim.command('echo runspec#SpecPath("bar/foo_spec.rb")').should ==
          "bar/foo_spec.rb"
    end
  end
end

Running the spec produced the following, rather thrilling output:

$ bundle exec rspec spec/runspec.vim_spec.rb 
.

Finished in 0.49066 seconds
1 example, 0 failures

As it ran, I saw an instance of Vim fire up and then be shut down as the test suite completed. More importantly, all my specified behaviour was checked without my having to interfere.

This was a great start but is only really good if you are testing functions that have no side-effects. However, one part of my plugin relies on checking the file system to determine its behaviour: namely, it reads a Gemfile.lock to determine whether to use rspec or not. How could we test this sort of functionality?

Well, we can simply write out a dummy Gemfile.lock during our test and see if the function works correctly:

File.open("Gemfile.lock", "w") do |f|
  f.puts(<<-EOF)
GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.1.3)
    rake (0.9.2.2)
    rspec (2.9.0)
      rspec-core (~> 2.9.0)
      rspec-expectations (~> 2.9.0)
      rspec-mocks (~> 2.9.0)
    rspec-core (2.9.0)
    rspec-expectations (2.9.1)
      diff-lcs (~> 1.1.3)
    rspec-mocks (2.9.0)

PLATFORMS
  ruby

DEPENDENCIES
  rake
  rspec
  EOF
end

However, this file won’t be removed at the end of your test runs and, what’s worse, it could potentially interfere with other tests.

To fix this, we can use Ruby’s Dir.mktmpdir and Dir.chdir in an RSpec around hook to create a temporary directory and cd into it for every single test. As we are re-using the same instance of Vim for each test, we’ll also have to cd the server into each new directory:

# At the top of your spec:
require "tmpdir"

# In your example group:
around do |example|
  Dir.mktmpdir do |dir|
    Dir.chdir(dir) do
      vim.command("cd #{dir}")
      example.call
    end
  end
end

By using the block form of both mktmpdir and chdir, Ruby will destroy the temporary directory and restore the current working directory after every run.

This made it possible for me to write specs like the following without worrying about the disk being littered with test files:

it "finds a test with the most similar name" do
  FileUtils.mkdir_p("test/unit")
  FileUtils.touch("test/unit/user_test.rb")

  vim.command('echo runspec#SpecPath("app/models/user.rb")').should ==
      "test/unit/user_test.rb"
end

At this point, there was quite a bit of set up in the spec file so I decided to move it out into a separate spec_helper.rb to keep the actual tests concise:

require 'tmpdir'
require 'vimrunner'

RSpec.configure do |config|
  config.around do |example|
    Dir.mktmpdir do |dir|
      Dir.chdir(dir) do
        VIM.command("cd #{dir}")
        example.call
      end
    end
  end

  config.before(:suite) do
    VIM = Vimrunner.start
    VIM.add_plugin(File.expand_path('../..', __FILE__), 'plugin/runspec.vim')
  end

  config.after(:suite) do
    VIM.kill
  end
end

Note the use of a VIM constant in before(:suite) instead of a simple local variable so that the server is available to all specs and the changing of after(:all) to after(:suite) so that the Vim server is shut down once the entire test suite is finished and not just those in the current example group.

So now we have a fully automated test suite that we can run locally; what next? How about continuously integrating that test suite to ensure that no commit breaks the build? How about using Travis CI for that purpose?

If you are unfamiliar with Travis, have a look at their Getting started guide which will lead you through creating an account, hooking it up to your GitHub repositories and running your first build.

In this case, we’re going to use their Ruby support to run our test suite. In order to do that, we need a default Rake task that will run the tests.

Firstly, we need to add Rake as a dependency to our existing Gemfile:

gem 'rake', '~> 0.9.2.2'

Then we need to create a Rakefile and a task to run the tests. Luckily, RSpec ships with such a task by default which we can use and define as the default like so:

require 'rspec/core/rake_task'

RSpec::Core::RakeTask.new(:spec)

task :default => :spec

(Note that RSpec::Core::RakeTask.new will name the task :spec by default but I’m being explicit here for clarity.)

You should now be able to run your suite like so:

$ bundle exec rake
/Users/mudge/.rbenv/versions/1.9.3-p125/bin/ruby -S rspec ./spec/runspec.vim_spec.rb
..............

Finished in 4.85 seconds
14 examples, 0 failures

All that is left is to configure Travis to do that too. Simply create a file named .travis.yml with the following contents:

language: ruby
rvm:
  - 1.9.3
before_install: sudo apt-get install vim-gtk
before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"

This will inform Travis to use the latest version of Ruby (as of writing) and to install a version of Vim with the necessary client-server functionality we need for testing. We also need to start a X virtual framebuffer so that we can create a Vim server successfully which is what the two before_script options are doing (see GUI & Headless browser testing for more information).

With that in place, the next time you push to GitHub, it should trigger a build on Travis: see Build #8 of runspec.vim for an example.

To see a fully worked example (and my finished plugin), feel free to browse the source code of runspec.vim and look at the build history on Travis CI.