The Pitchburgh Diaries - decoupled Layout Builder Sprint 1 & 2

Highlights

Let's start with a quick overview of what we've been working on:

• Evaluation of JSON:API
• Design of API and Open API specification
• Progress towards persistence layer
• Local development setup
• API for formatters in React
• API for Block plugins in React
• API for Layout plugins in React
• Evaluation of React drag and drop libraries
• Drag and drop support for blocks and sections

Background

You may also be wondering what this is all about!

In case you missed it, at DrupalCon Pittsburgh we successfully pitched to build a decoupled version of Layout Builder powered by React. In addition to that initiative, our team has been working on bringing editorial improvements to Layout Builder in core. My colleagues Daniel Veza, Adam Bramley and Mohit Aghera have been working on this during our internal contribution time and company-wide innovation days.

As part of this, we've been collaborating with key contributors including the subsystem maintainer Tim Plunkett and Drupal core's product manager Lauri Eskola.

Lauri has been doing user research into Layout Builder pain points and trying to frame the long-term direction for Drupal's page building experience.

Lauri was able to take us through some wireframes he's been working on in this space and these aligned well with our plans for a decoupled layout builder.

I encourage you to review these and provide feedback in the core ideas queue issue.

So, now you know the background, let's get into a summary of what we achieved in the first two sprints.

JSON:API analysis

One of the deliverables for our bid is to create a persistence layer in Drupal to allow retrieving and updating the layouts from the decoupled editor.

With JSON:API in core, this felt like a natural starting point. We spent time reviewing the current patch to add JSON:API support to layout builder. There are some issues with this patch as follows:

• It contains a security issue which we've left a review on; and
• It doesn't comply with the JSON:API specification regarding relationships

The JSON:API specification states that relationships must be present as a top-level field in a resource object. This is so related content is addressable.

With layout builder data there are a number of related resources as follows:

• Sections
• Components (block plugin instances)
• Any referenced content these block plugins refer to. For example, the inline block plugin references a block content entity. A field block plugin might reference a media entity from an entity relationship field.

The current patch just emits the section and component data as is and does not provide a way for these relationships to be surfaced at root of a resource object under a relationships key.

Complying with the specification

So if we were to take a step back and think about how we would provide Layout Builder data and comply with the JSON:API specification – how would it look?

Going back to the data model we would need to allow addressing sections and components.

This would allow us to have a 'sections' field in the top-level resource as a relationship.

However, at present sections in Layout Builder data do not have identifiers. There is an open issue to add this but that's only part of the issue. The larger issue is that if we needed to fetch a section resource, we have no way of retrieving it by ID. This is because Layout Builder data is stored in a blob inside a SectionList field on the entity to which the layout is attached. To achieve that we'd then need to maintain some sort of 'component index' table. This would allow us to retrieve the layout entity based on a section ID. We would probably have to vary each section resource based on the layout plugin, just like we do for nodes by node-type. For example, we'd have a section--onecol and a section--two-col resource.

So let's say we had that. The next issue we have is we need to be able to address components. These do already have a UUID, but again that information is only stored in the field blob. That means we'd need to extend our 'component index' idea to also support looking up a component.

Then from there, we need to be able to distinguish between fields/properties on a per-component basis. For example, an inline block component has a reference to a block content entity via a relationship. Other components (block plugins) don't. We would therefore need to vary resource types by block plugin ID. As a result, we might have component--inline-block and component--field-block--node--field-media.

With all those pieces in place, we should be able to achieve addressable resources for all of the related data in Layout Builder. However, the query to retrieve all the information we need would be quite involved, as it would need to traverse from node to section to component to block content or media.

All of this is only thinking about retrieval. We also need to be able to persist changes, particularly to block content for inline blocks. When a layout is updated we would need to write changes to the block content, then to each component, then to each section and finally to the entity the layout applies to. Each of these would be a separate HTTP request. What happens if one of those requests fails? How would we recover from that state? JSON:API has already thought of this issue and has an atomic extension in the specification.

However, there is no existing implementation of this extension for Drupal, so we'd also need to write that.

Considering all this work and the limited size of the budget for this project, we decided that JSON:API isn't a good fit at this time. We intend to open a meta and child issues for each of the discreet pieces of work that would be required to build a JSON:API compliant endpoint for Layout Builder.

Specifying our own API

As we decided not to use JSON:API, we were able to design an API specifically tailored to the work we're doing. But we need to make sure to document it. Open API is the best way to do this as we can auto-generate rich documentation with tools such as Swagger UI.

We have finalised the API specification in the Decoupled LB API module on Drupal. Oh, and did I mention that Gitlab automatically turns this into interactive documentation with Swagger UI. 😮😍

We have started work on implementing this and have working versions of the following operations including test coverage:

• getSectionsAsJson
• getBlocksAsJson
• getLayoutAsJson

Part of this involved some key decisions around how block plugins in Drupal will interact with React. More on that later.

Collaboration with the Gutenberg pitch

There is some overlap in the work we're doing with the Gutenberg project, another Pitchburgh grant winner. As a result, we've been catching up with that team to ensure we aren't duplicating effort.

One such issue we identified was that both projects will need a shared way to load React from Drupal. At present, the Gutenberg module includes a transpiled version of React and React DOM and is loading these from a libraries definition.

As we will also need to do similar and we don't want to end up with two versions of React loaded from Drupal, we identified that we'd need a common shared module to do this.

We approached the maintainers of the Drupal 7 React module and asked if we could be added as maintainers to use this namespace for that purpose. The maintainer David Corbacho was happy to oblige. Thanks, David!

Local dev setup

Our next step was to get a local development setup that allows Front-end developers to contribute to the project without needing to setup Drupal. This includes tooling that front-end developers expect including:

• HMR with Vite
• Mocking of APIS with msw
• Storybook setup
• Typescript setup
• Jest and React Testing Library setup
• Prettier/eslint setup

Setup of redux for state management

To keep track of the layout state, we chose Redux to support global state management. One of the features present in Lauri's wireframes is Undo and Redo support. Keeping track of state with Redux means this will be simple to implement when the time comes. Using redux also allows us to separate state manipulation from the UI and this means we can easily write unit-tests without needing to simulate interaction.

Selection of Drag and Drop library

A key tenet of the layout editing experience is the ability to drag and drop blocks (components). Currently Drupal core supports moving blocks, but not moving sections. There is an active issue to add support for re-ordering sections. One of our goals is to support that functionality from the onset.

Drag and Drop in React is not unique to this project so we did an evaluation of existing React drag and drop libraries. The two we focussed on were react-dnd and react-beautiful-dnd.

After reviewing both packages we decided to go with react-beautiful-dnd. It features keyboard control, auto-scrolling of drop containers and screen-reader support. It was written by Atlassian (creators of Jira, Confluence etc). The only item of possible concern here was that it is no longer under active development. It is however still doing releases for security issues etc. We rejected react-dnd because of its lack of built-in support for keyboard control, screen reader and scrolling. There is also an active security report in its issue queue that has not yet been responded to. 

If we get into issues with the 'no further development' status of react-beautiful-dnd, there is a sympathetic fork, however, it hopefully won't come to that. Another plus in the react-beautiful-dnd column is that it was used by puck, an existing React-powered layout editor. We evaluated that as part of this project and it provides some great inspiration, but doesn't fit with Drupal's data model or the longer-term goals for in-place editing seen in Lauri's wireframes.

After selecting this we set about building a prototype using it to validate the API and have dragging blocks and sections working.

Data model design in React

One of the driving factors behind the design of the layout builder in React is it must map back to existing concepts in Drupal. We're building a layout editor, but Drupal will still need to be able to render the layout in the front end once the layout is saved.

To manage this we've designed an API for React components that is familiar to concepts in Drupal. We have built proof of concepts of Block plugins, Layout plugins and Formatter plugins.

We've built a hook system to support retrieving entity view display information client-side.

To validate all of this we built an InlineBlock React component that maps to the InlineBlock plugin in Drupal core. The video above showing blocks being dragged around makes use of these components. Each component returned from the API includes a plugin ID and in the proof of concept these map to a React component. 

In the example, you see the InlineBlock component taking the values from the API, reading entity view mode information and using that to hand off rendering to formatter plugins. Similarly, the API returns sections using their Drupal plugin ID. These are mapped to React components so a one-column component handles the one column seen in the example and a two-column one handles the two-column component.

This ensures that each component is in control of its markup, just like in Drupal with Twig. Regions are handled by a Region component that abstracts away the drag-and-drop functionality. Each Layout plugin just needs to emit the region and even has control over the markup. In the example, the right column of the two-column layout uses an aside element.

Here's how that looks in code:

import React from "react";
import { LayoutPluginProps } from "../../state/types.ts";
import Region from "../LayoutEditor/Region.tsx";

export const Preview: React.FC<LayoutPluginProps> = ({
  section,
  sectionProps,
  regions,
}) => {
  return (
    <div {...sectionProps} className={`section section--${section.layout_id}`}>
      <Region
        regionId={regions.first.id}
        className="layout__region layout__region--first"
      />
      <Region
        regionId={regions.second.id}
        as={"aside"}
        className="layout__region layout__region--second"
      />
    </div>
  );
};

export const id = "layout_twocol_section";

For those of you familiar with a layout twig template in Drupal, this will look pretty familiar. You are given some section props to output on the outer container, just like the attributes you get in twig. You are given a map of region data and you can place the Region component where you need it, the only requirement is that you pass along the ID in the regionId prop. This isn't that different from the region_attributes variable in twig.

All of this is booted from a single LayoutEditor component that takes a registry of block, layout and formatter plugins. As work progresses this will expand to include widgets too.

Each of these registries is a map where the keys are the plugin IDs and the values are a promise that will load the component. In our local development setup, we're using Vite's glob support to automatically load all plugins in a given folder. Our thinking is in the Drupal space we will use an alter hook to add additional properties to plugin definitions and then use drupalSettings to emit a mapping of plugin IDs to file paths. We will be able to use the built-in import function with the file paths to provide the promises. 

So for example, a hook block alter will add a property to each block plugin for the file path of that equivalent React component. This will allow modules and themes to alter that definition and swap out a component provided by default with their own implementation in React that is specific to their project. We will collate all of these when booting the LayoutEditor in Drupal and pass that via drupalSettings into the entry point. We will do something similar for the other plugins.

Approach for normalizing section components

All of this also relies on allowing block plugins in Drupal to have control over the data that is sent in the API. For instance, in the inline-block example, it needs to be able to pass along the block content entity's field values for rendering/editing sake.

The current API handles this using Drupal's existing serializer. The decoupled LB API module adds a normalizer for SectionListInterface objects (how layouts are modelled in Drupal). This loops over each component and normalizes an instance of the relevant block plugin. The default block plugin normalizer just extracts the block configuration. However, due to how Drupal's serializer works, any module can add a new normalizer that targets a more specific class than BlockPluginInterface and give it a higher priority, then modify how the data is sent.

To support working with inline blocks, we've added a normalizer for the InlineBlock plugin that intersects the block content fields with that of the configured view mode and passes those along. In the example above you can see we're rendering the body field of the block. This will also provide us with the implementation point for dealing with persisting updates. We will be able to mutate these properties in the layout as edits occur and send them back to Drupal. The normalizer will be able to denormalize the incoming values and put them back in the format Drupal expects so that saving the layout works the same as it does now.

Next steps

We've made significant progress in the first two sprints. Our high-level goals for next sprints are:

• Finalising the persistence layer in line with the Open API specification
• Adding support for widgets and editing

Thanks

Thanks to my colleagues at PreviousNext who've been a great sounding board in validating some of these design decisions, in particular Daniel Veza who has helped to workshop many of these ideas.