Speeding up Vagrant

A little bit of background: We use Vagrant to replicate our Production environment on each one of our developer’s machines. We’ve gone through numerous configurations trying to figure out which one would work the best. A couple of things have stayed largely the same however - we use the VirtualBox provider and the chef recipes we use to set up those VMs are the same we use in production.
Originally we used chef-server to provision each of the machines - this meant our Devs were tied to our corporate office. Every single run required a connection to our internal Chef Server, and that sort of cramped our style! We are aiming to simplify the setup required to get a developer going, so the less stuff we need on each machine, the better.

Enter the next phase: Chef Solo! At the beginning we were using berkshelf, as well as some custom scripting to ensure that older cookbooks got purged from the machine cache. This stuff was all loaded automatically on every vagrant command ran in our project folder. Eventually we transitioned to the vagrant-berkshelf plugin. This allowed us to get rid of some of our custom tooling, and it made setting up the development environment easier - no more berkshelf installed on the host and no more chef either, which meant no more worrying about ruby environments.

Fast forward to the future, and now we no longer have anything berkshelf related installed on our dev machines. At the time of this writing, vagrant-berkshelf is not supported with Vagrant 1.5.1, so we were forced to come up with a custom solution. I think that we ended up finding an even better solution! We have jenkins running a job that essentially does berks package to tar up all of our cookbooks and their dependencies in one place, and then we send that artifact down to each one of our Devs. The code looks something like this:

artifact = 'http://ci.example.com/view/artifact/package.tar.gz'
unless Dir.exists?('/srv/vagrant/cookbooks')
  Dir.mkdir('/srv/vagrant/cookbooks')
  Dir.mkdir('/srv/vagrant/berks')
  `curl -o /srv/vagrant/berks/package.tar.gz #{artifact}`
  `tar -zxf /srv/vagrant/berks/package.tar.gz -C /srv/vagrant/cookbooks`
  FileUtils.rm_rf('/srv/vagrant/berks')
end

def up?(site)
  Net::HTTP.new(site).head('/').kind_of? Net::HTTPOK
end

if (Time.now - File.stat('/srv/vagrant/cookbooks').atime).to_i / 86400.0 >= 1 && up?('ci.example.com')
  FileUtils.rm_rf('/srv/vagrant/cookbooks')
  Dir.mkdir('/srv/vagrant/cookbooks')
  Dir.mkdir('/srv/vagrant/berks')
  `curl -o /srv/vagrant/berks/package.tar.gz #{artifact}`
  `tar -zxf /srv/vagrant/berks/package.tar.gz -C /srv/vagrant/cookbooks`
  FileUtils.rm_rf('/srv/vagrant/berks')
end

I still need to account for different situations (exits during this process mess up future vagrant runs unless you manually clean things out, etc) but this is functional for now!

That shaved a decent amount of time off a vagrant up because now the hard work has been offloaded to CI!

We also started using vagrant-cachier, and I can’t believe we got by without it before! It shaves about 2 minutes off of a vagrant up by caching our apt, npm, composer, and gem installs. In addition to being quicker, this also saves us quite a bit of bandwidth!

We still use vagrant-omnibus, which is a great plugin, however it does take time to install Chef each time you bring a box up. At this point I consider that a neccesary evil. Installing it every time ensures we are up to date. Having to repackage the box every so often with new chef versions would take a lot more effort (not that vagrant package is terribly difficult)!

The other change we made was to use NFS for shared folders in vagrant instead of the default VirtualBox shared folders. Mitchell Hashimoto wrote an excellent article detailing the performance differences between shared folder types. With vboxfs out of the way, we no longer had to use vagrant-vbguest to ensure the guest additions were up to date. Performance of our projects went way up with the switch as well.

NFS brings some special challenges of its own though, as it mounts folders with the same uid as the host user. Our solution for this is to dynamically assign a uid to our deploy user that is the same as our host user (using Process.uid in the vagrantfile) and then also setting that to the uid of www-data and the redis user. This allows us some flexibility with the mount and permissions. Obviously that’s not what we do in production, but it is close enough™. We also has some issues with chef - on directory and template creation it attempts to chown, and nfs does not appreciate that with its default settings.

In future posts I’ll go into a little bit more specific detail about our NFS setup, and I might also go into the steps a new Dev has to take with their machine to get things going.