Monitoring Drupal Site Status at Scale

Drupal has had the ability to send email notifications when security updates are required. However, if you have a large volume of sites, there is a high signal to noise ratio, and often a critical security update will get missed.

Existing Solutions

There are a number of existing solutions out there. Why would we build our own? 

After an evalutaion, there were a number of compelling reasons to roll our own:

• We wanted to push the check data rather than open up each of our sites to an external query;
• The data is very sensitive as it could expose security vulnerabilities, we wanted to ensure we had control over it;
• We wanted to support Drupal 7 and 8 as well as a number of other bespoke PHP applications we support. The solution needed to work with and without Drupal.

 

Our Solution

Our solution to the problem was to create a centralised Status Dashboard API which collects information from each site, and provides an API to query that information.

Checkers and Dashboard Connector

Each site has a number of configured checkers which are triggered when cron runs. The checks returned are aggregated together and pushed to the Status Dashboard API as a snapshot of the status at one point in time.

Each check stores a type, a name and an alert level. Alert levels are: notice, warning, and error. Checkers are simple CTools plugins which implement CheckerInterface which has a single method  getChecks() to return an array of check data.

Current checkers include:

• Module Status: whether a normal (warning), or security update required (error), or whether the module no longer supported (error).
• Requirements: checks if there are requirements that are not being met. These are usually displayed in the Status Report admin page.
• Performance: checks that page cache, block cache, and CSS and JS aggregation are all enabled.
• Disabled Modules: checks whether there are enabled modules which should be disabled. (e.g. Devel, Views UI, Field UI)
• Enabled Modules: checks whether there are disabled modules which should be enabled. (e.g. Paranoia)
• Theme: checks whether the 'Rebuild theme regisitry on each request' flag is enabled
• Views: checks whether there are any views that are overridden. (Your config is in code, right?)
• Admin Views: checks if the dangerous 'user delete' action is available
• PHP version: checks if we are on a supported version of PHP

It is easy to implement your own by creating a CTools checker plugin.

Status Dashboard API

The dashboard API is a simple REST web service built on Silex. Sites push their data with a unique key, and require authentication over secure HTTPS. The snapshot data is stored in a MySQL database, but treated as ephemeral, it does not matter if it gets deleted, as it will get re-populated with a new snapshot the next time cron runs on a site.

The API allows you to view all snapshots for all sites as one big chunk of JSON. Each snapshot provides a summary of the number of alerts by level. This makes it easy to find sites with error alerts quickly. You can also apply a filter to show only sites with an error alert.

You can view the snapshot details for a specific site keyed by its unique site ID to see each check and its alert level. You can also filter these to show only the checks for a site which have an error alert level. This allows you to easily find which checks are causing all the problems.

Command Line Interface

Of course, trying to read JSON regularly isn't much fun. So we built a simple Command Line Interface (CLI) tool called Dashboard Console using Symfony Console to do the work of querying the Status Dashboard API and returning the results in a nice human-readable format.

A sample output for getting all site snapshots is the following:

Here's the output for querying a specific site: 

Open Source!

We're happy to release two of these components: Dashboard Connector and dashboard-console, as open source. You can download Dashboard Connector from Drupal.org at https://www.drupal.org/project/dashboard_connector while the Dashboard Console CLI utility is available on Github at https://github.com/previousnext/dashboard-console

Of course, you will need to build your own Dashboard API, but that will be simple enough once we publish the API documentation.

 

The Future

For the first round of development, we took an API-first approach, supported by simple command line tools. This gives us the benefit of being able to spin something up quickly, and gives us agility to make changes when necessary. Possible future development work could include supporting alternate authentication methods, and a web-based interface for view snapshots, and querying them.