Introducing Symfony Messenger integrations with Drupal

The SM project brings Symfony Messenger and its ecosystem together with Drupal.

Symfony Messenger is a powerful alternative to Drupal’s @QueueWorker, enabling real-time or precise execution of scheduled tasks. It’s also a viable replacement for hook_cron enabling you to schedule dispatch and processing of messages according to crontab rules, again in real-time, rather than waiting for server-invoked or request-termination cron.

Messenger can be used as a user-friendly alternative to Batch API, as the user's browser is not blocked, while integrations such as Toasty can be programmed to notify when the last of a “batch” of messages completes, communicating to the user via user-interface toasts.

The Drupal integration includes additional niceties, such as intercepting legacy QueueWorkers for processing data through the Symfony Messenger bus, and end-user UI notifying a user when tasks relevant to them have been processed.

During this and the following series of posts, we’ll be exploring the benefits of real-time processing and user-friendly features that improve the overall experience and outputs.

Messenger

First up, we’ll cover the main features of Symfony Messenger and how it works.

As a developer working with Messenger, the most frequent task is to construct message and associated message handlers. A message holds data, while a message handler processes the associated data.

A message is inserted into the bus. The bus executes a series of middleware in order, each of which can view and modify the message.

If a transport is configured, the message may be captured and stored for processing later.

Typically the bus, middleware, and transports are configured in advance and rarely changed. Message and message handlers are introduced often without needing other configuration.

• Message — an arbitary PHP object, it must be serialisable.
• Message handler — a class that takes action based on the message it is given. Typically a message handler is designed to consume one type of message.
• Middleware — code that takes action on all message types, and has access to the containing envelope and stamps.
• Bus — a series of middleware in a particular order. There is a default bus, and a default set of middleware.
• Envelope — an envelope contains a single message, and it may have many stamps. A message always has an envelope.
• Stamp — a piece of metadata associated with an envelope. The most common use case is to track whether a middleware has already operated on the envelope. Useful when a transport re-runs a message through the bus after unserialisation. Another useful stamp is one to set the date and time for when a message should be processed.
• Transport — a transport comprises a receiver and sender. In the case of the doctrine database transport, its sender will serialise the message and store it in the database. The receiver will listen for messages ready to be sent, and then unserialise them.
• Worker — a command line application responsible for unserialising messages immediately, or at a scheduled time in the future. Messages are inserted into the bus for processing.

The stars of the show are buses. One bus is ready out of the box, which comprises a series of ordered middleware. A message is dispatched into a bus, where each middleware has the opportunity to view and modify the message (and its envelope). It's unlikely you’ll need to think about middleware, as the default set may already be the perfect combination.

When a message is dispatched to a bus, you can choose to wrap it in an envelope and apply stamps like the DelayStamp . A message will always be wrapped in an envelope if you don’t do it explicitly.

Buses have a series of default middleware. The main middleware to note are the transport and message handler middlewares. When a transport is configured for messenger, the transport middleware will capture the message, serialise it, and store it somewhere. For example, a database in the case of the doctrine transport. Any middleware after the transport middleware are not executed, for now.

When running the worker, you are opting to choose which bus and transport to run. The command will listen for messages as they are stored, and if the time is right, messages will be unserialised and inserted into the bus. The message will begin its journey yet again, iterating through all the middlewares from the beginning. When the transport middleware is hit, it will detect the message has already been in the transport to prevent recursion. This is done by checking the ReceivedStamp stamp added to the message envelope.

Transports: synchronous, asynchronous

Out of the box, when a message is dispatched into the bus in a CLI or web request, it will be processed synchronously. All middleware will operate on the message in a set order, including the message handler middleware.

The greatest advantage of using Messenger is the ability to asynchronously handle messages outside of the thread they were originally dispatched. That is: asynchronously. This can be useful for improving the web request response times and reducing the memory usage and limit of web requests (allowing for more FPM threads on a machine). Bulky business operations that would typically, or should be, constrained by the limits of the web thread have more breathing room. A CLI runner/container may be set up with a little more memory and processing capability with the explicit direction to listen for messages and handle them in real-time, either as soon as possible or as scheduled.

Upcoming posts in this series will dive into aspects of Symfony Messenger and SM:

• Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
• Real-time: Symfony Messenger’ Consume command and prioritised messages
• Automatic message scheduling and replacing hook_cron
• Adding real-time processing to QueueWorker plugins
• Making Symfony Mailer asynchronous: integration with Symfony Messenger
• Displaying notifications when Symfony Messenger messages are processed
• Future of Symfony Messenger in Drupal

The next post covers the implementation of a message and message handler, and a comparison with Drupal core’s @QueueWorker plugins.