Drupal Testing Roadmap

Recently the patch to bring Mink based testing to drupal core went green. As result of that Lee Rowlands (@larowlan), Nick Schuch (@wesome1989), Adam Hoenich (@djphenaproxima), and myself (@grom358) had a discussion to create a roadmap for improving testing in Drupal core. Here is what we discussed.

BrowserTestBase

BrowserTestBase is the new test base class that has been created in the patch that builds on top of Mink and PHPUnit. This is mostly complete and is aimed to be the first thing to be committed to core. Its aim is to be a replacement for WebTestBase so can move browser based testing off the custom code in Simpletest. This was the central topic of our The Future of Functional Testing in Drupal talk at Amsterdam DrupalCon. With the move to BrowserTestBase we will be leveraging the defacto standards of Mink and PHPUnit instead of code that is only being used by the Drupal project.

Real Browser Tests

With the move to Mink we can look to bring actual browser testing to Drupal with support for the full webstack including JavaScript. Currently Simpletest emulates a very basic browser that understands Drupal AJAX commands, so any model that relies on a rich web UI are left in the dark in regards to testing. Using the power of Mink can incorporate testing of rich web user interfaces. In our discussion we proposed PhantomJs since it has a good tradeoff between features and performance. Before this can happen we need the new test bot infrastructure from the Drupal CI team.

KernelTestBase

Sun (@sun) has a patch to bring KernelTestBase to PHPUnit and is in need of a reroll. BrowserTestBase should share codebase with KernelTestBase in the future. The only difference being BrowserTestBase installs Drupal whereas KernelTestBase just gives you a bootstrapped kernel to work with. KernelTestBase also doesn't require running tests through a web server. Simple browser tests (ie. no JavaScript) could also bypass a web server by implementing a custom mink driver on top of BrowserKitDriver that talks to the kernel directly. This should result in performance improvements as it removes networking overhead of running tests.

One issue that needs resolving here is removal of exit() calls in Drupal as they prevent being returned a Response object. Most of these are already gone out of d8, but there still some to go.

Convert Tests

Drupal 8 has moderinized Drupal and is now taking advantage of modern object oriented PHP, which brings with it unit tests. For anyone that has had experience with Simpletest you will know how long it can take to run tests. The reason being functional tests need to setup a Drupal site and go through the full Drupal stack. With the move away from Simpletest brings with it a great opportunity to rethink how we test in Drupal. We already started to use unit tests and there been more focus on API (which allows for things like headless Drupal). This means we can use more unit testing, or kernel testing instead of relying on the slower functional tests; enabling faster feedback loops when developing Drupal.

Removal of run-tests.sh

Once tests have been moved to PHPUnit we can replace having custom test runner (run-tests.sh) with phpunit. And use tools like Parallel PHPUnit. PHPUnit results output is a common format and supported by a wide variety of tools. The current Drupal.org test infrastructure is heavily tied to run-tests.sh at the moment, but a new testing infrastructure is underway and will fit in nicely with the work to improve the testing codebase.

Summary

By moving to PHPUnit we leverage a testing framework that is widely used in the PHP community and feeding improvements both upstream and downstream from a larger community. There we be less code to maintain in Drupal core and allow us to focus less on testing frameworks and more on writing tests for our code.