Automated Drupal Testing with Github Pull Requests

The Process

Our git branching model following something similar to Gitflow though slightly simpler. In essence, we have a branch per story or feature, and merge into a stable master branch, before being deployed to a QA server.

 Before it can be deployed, it needs to jump through a few hoops.

Automated Tests

 If we are going to fail, we want to fail early. First we check for code style. While this may seem nit-picky, it ensures that we are at least using the same white-space conventions, but can quickly catch missing function and file comments. We can also add in some of the other PHP Quality Assurance Toolchain at this point, such as checking for code size bloat, unreachable code, and copy and paste detection. These tests are very fast, and don't require bootstrapping Drupal or a database.

Next are Unit tests. These are typically done using DrupalUnitTestCase which, again does not require a database. These are used for testing the logic of a particular function in isolation from the rest of the system. For example, a simple calculator function can be tested with a whole range of inputs and their expected outputs. This doesn't require an expensive page load in order to verify it works as expected, so these tests can be run quickly.

We are also looking at using PHPUnit for unit tests due to the rich feature set it offers, such as mock objects, code coverage reports, and good IDE integration. PHPUnit is planned on being added to Drupal 8, and will become more important as Drupal's code becomes more object-oriented, decoupled and easier to test with unit tests.

Lastly, we use DrupalWebTestCase for functional testing. This allows us to test sites as if we were using a browser. This tests end-to-end integration, including the database, a Solr server, or an external web service we are integrating with. These tests are typically slow, and require an environment that is close to the production environment.

All of these automated tests comprise the build. If one step fails, the entire build fails.

Jenkins

 Jenkins is a great tool for automatically running a build. You can tell it to watch a git repository for changes, and if a change is detected, to fire off a build. The build can be anything, but in our case its running the tests suite. This is also known as continuous integration, because it happens continuously, not on a scheduled nightly build.

Because we want to get feeback as quickly as possible, we run the code style and unit tests locally on the Jenkins server.

However, Simpletests require a working Drupal environment, and each project tends to have different requirements, so we use Jenkins to run a Capistrano Drupal deployment of the project, and run the tests remotely. This allows us to have tests that test integration with 3rd party web services, or Solr for example. The test results are then returned to the Jenkins server, and notifications sent to the team.

Github Pull Requests

Github pull requests are a great way to manage peer review of code, especially in a distributed team. Developers are able to post inline comments on each line of code. Issues raised can be addressed by subsequent commits and contributions from other team members until the feature is just right.

Pull request peer reviews are an important part of the process. They ensure that developers are using the appropriate techniques, have clear, concise and well documented code with appropriate levels of test coverage.

Putting it all together

 The important piece in this puzzle, is the github pull requests builder plugin for Jenkins. Great title. And that's exactly what it does.

Instead of running your build when code is merged, this plugin integrates with Github's pull request feature, that builds the project on that branch as if it had been merged already. This is important, as it confirms that the code will not only merge cleanly, all tests will pass once it has been merged. This is our way of ensure the master branch is always stable.

Here are some screenshots to give you an idea of the Github pull request build status indicators.

The test is buiding, we're not sure yet it it's passed, so don't merge yet!

The tests have passed, we're ok to merge now!

Summary

The workflow is:

• You create a branch for your new feature
• You push it up to a github remote branch when you are ready to share it
• You create a pull request
• The Jenkins build is fired off
• If successful, the pull request merge button is green. You're good to go.
• If not, the merged button is grey. Use caution!
• Another developer on the team does a code review (on fully tested code)
• If further changes are needed, a simple comment 'test this please' in the pull request comments will fire off a new build
• Review completed, and code merged into the stable branch

Once the code is in the stable branch, a new release can be deployed to the QA server for manual testing. This includes anything that cannot be automated, such as accessibility tests, cross browser tests, etc. We use this as a final QA process to ensure the feature has been delivered and meets the goal of the original story.

Adding automated tests to Github pull requests can make a big difference to the quality assurance of your project, and is well worth the effort of setting it up.

For more background information, check out larowlan's session on Writing Automated Tests for Drupal and boztek's session on Real world continuous integration for Drupal at the recent DrupalCon Sydney 2013.