Architecting DrupalCI at DrupalCon Amsterdam
At the recent DrupalCon Amsterdam sprints something amazing happened, people from all corners of the globe assembled to sprint on DrupalCI. DrupalCI is an initiative born out of the requirement for new testbot infrastructure. Our goal is to implement a brand new Continuous Integration (CI) workflow that can not only be used for Drupal but anyone wishing to run a CI infrastructure / Automated tasks. Until this point we had only corresponded via a weekly hangout and IRC.
While this was keeping us on track with building out some of the components, the conference gave us an opportunity to sit down in the same room and perform an end-to-end architectural review to ensure we didn't have any gaps. A modular design approach has been used to ensure that many of the following components could be used as a standalone entity in any infrastructure.
The components
Drupal.org integration module
This component will be used to trigger job runs on the DrupalCI infrastructure, in response to new patches and/or commits occuring on drupal.org (or git.drupal.org).
API
At the top of the DrupalCI stack is a Silex application which acts as an interface into the infrastructure . The API is responsible for passing build requests into the dispatcher, and build outcomes back to the results site. Since this component is abstracted away from the dispatcher, we can swap out the backend with whatever new technology comes out in the coming years. Long term this component could also provide functionality such as "streaming build output" so you can see the console output in real time.
Dispatcher
The dispatcher is in charge of handling the resources for each build. For this role we have chosen Jenkins. Jenkins is a well known CI project with may plugins available for easy extensibility. One plugin of note is the AWS EC2 plugin, which we use to spin up AWS AMI images with a preloaded testbot runner, containers and the results site console application.
Job Runner
The Job Runner is the core of the build as it orchestrates what steps are performed. The job runner comes in the form of a number of job-specific scripts, and a Symfony Console application which serves as both the automated test runner and manual end-user interface for standalone local testing. For standalone use, the job runner was also "in the wild" at DrupalCon Amsterdam sprints. Huge kudos to Ricardo Amaro with his work on this component!
Results
One component which had not recieved much attention until the DrupalCon Amsterdam sprints is the results server, responsible for archiving and exposing test results and build artifacts for all DrupalCI jobs. This server will be a Drupal 8 site with a single content type, allowing us to push artifacts and render them in interesting ways to the end user. Our base development server has been built out on Drupal.org infrastructure, and we are now ready to begin prototyping the actual build. We are still working on a backlog, but given it's simplicity it won't be hard to prototype out in the mean time.
Metrics (monitoring / logging)
Last but not least is the Metric component. We want to ensure our CI infrastructure is up and running, as well as pulling some interesting statistics (want to know how many builds etc we got at a Drupalcon?). We will use this component for both monitoring the availablity of the architecture and keeping logs for a set period of time.
Putting it all together
Call to arms
With all of these components identified, and the outstanding gaps now mapped, there is no better time to join in on all the fun. The goal is to have a maintainer for each of these components, that way we can distribute out the effort while also giving everyone a chance to "own" there portion of the infrastructure. With that in mind please don't hestitate to contact myself, Jeremy, or Ricardo if you want to get involved in this very interesting project.
Contact details
Weekly Hangout: http://bit.ly/1sBysAN
Timezone for the Weekly Hangout: http://www.timeanddate.com/worldclock/fixedtime.html?msg=Modernizing+Testbot+Hangout&iso=20141012T14&p1=210&ah=1
The following are our usernames IRC, we can be found in the channel #drupal-testing.
Jeremy Thorson - jthorson
Nick Schuch - nick_schuch
Ricardo Amaro - ricardoamaro
Tagged
Drupal System Administration