At the Drutopia project, one of our big focuses has been improvements to configuration management in Drupal 8. In this series, I'll be covering our work to date along with related efforts and contributions.
Drutopia is a platform cooperative initiative, building out cooperatively owned and hosted Drupal distributions. In our 2016 white paper, we recognized that the Configuration Management Initiative (CMI) in Drupal 8 "produced a lot of improvements in configuration handling" while noting that these "mainly addressed the use case of 'staging' configuration from one version of a site to another, a site-building technique that lower budget sites often don’t have time or money for." We committed to focus on "the free software use case left out of Drupal core: reusable configuration that can be shared across multiple sites". For background, see Drupal 8 configuration management: what about small sites and distributions? and sections on Drupal 8, corporate influence, and the CMI in this interview.
There's a current initiative to improve configuration management in Drupal core. Dubbed "CMI 2.0", the effort comes out of a similar conclusion that limitations and missing use cases in configuration management are a major barrier to Drupal 8 adoption; see Angie Byron's post proposing the initiative.
In the past three years, we at Drutopia have contributed to a growing collection of Drupal plugins that together address some of the tricky problems involved in managing shared configuration. As well as in kind contributions by Chocolate Lily, some of our work was sponsored by Agaric and the National Institute for Children's Health Quality (NICHQ) to meet their needs for an in-house platform of community sites.
Just what do we mean by managing shared configuration?
Say I have a site built on a Drupal distribution that's for community organizing. I installed the site a month ago and got groups-related configuration such as a group type. Then I made some modifications of my own. I've just downloaded a new release of the distribution, including enhancements to the groups-related configuration. How can I update my site so that I have all the latest changes from the distribution--while still retaining any customizations I made? That's the key question we've tried to tackle.
A more abstract way of putting the problem is: how can we provide packages of shared configuration in a way that lets site administrators both customize their sites and merge in configuration updates?
This series will cover distinct aspects of the problem of managing shared configuration packages and, along the way, highlight specific solutions we at Drutopia have sketched in. Our efforts are very much works in progress. We're not sure we've even got all the problems right, let alone fully addressed them ;) But have we made progress? Yes, we have. By sharing it here, we hope to raise the profile of these problems and solutions and invite further perspectives and contributions.
Here are the aspects we'll cover, along with associated Drupal modules:
- Configuration Providers (Configuration Provider)
- Configuration Snapshots (Configuration Snapshot)
- Respecting Customizations (Configuration Merge)
- Configuration Alters (Config Actions, Config Actions Provider)
- Updating from Extensions (Configuration Distro, Configuration Synchronizer, Config Filter)
- Packaging Configuration (Features)
- Base Configuration (Configuration Share)
- Summary and Future Directions
As we go, we'll briefly introduce some relevant concepts in Drupal 8 development as they come up, while pointing to sources of further reading.
First up, configuration providers!
When you install a Drupal extension (a module, theme, or install profile), you'll often get configuration installed on your site. That's because the extension provides that configuration. Technically, the configuration is sitting in files in one of two directories within the extension.
Files that are in
config/install are installed when the extension is installed. There's also optional configuration: files in
config/optional that are installed when certain conditions are met. For details about how this works, see the documentation on drupal.org.
This system works great for the use case Drupal core supports: one-time installation of configuration provided by extensions. But when it comes to updating extension-provided configuration, we run into problems.
In order to run configuration updates, we need to know what updates are available. For extensions, this comes down to the question: what configuration would be provided if the currently installed modules were reinstalled?
Because core is tightly tied to the model of one-time installation of extension-provided configuration, relevant code is not available in a form we can use. A key problem is with optional configuration. In core, the logic to determine what optional configuration will be installed is embedded in
ConfigInstaller::installOptionalConfig(). But this method doesn't just answer the question of what will be installed--it also goes ahead and installs it. Ouch.
Beyond optional configuration, a second issue is that core's code is limited to the two use cases covered by
config/optional. In practice, though, contributed modules define configuration-providing directories of their own to address additional use cases. Examples of such are the Configuration Share module, which provides a
config/share directory, and the Config Actions module, which enables programmatic alteration of configuration through actions provided at
config/actions. (More on both of these modules later in this series.)
So, as one part of the puzzle of how to update configuration from modules and themes, we need a solution that can support configuration providers, both those in core and any that contributed modules might provide.
The Drutopia-supported Configuration Provider module addresses this use case by providing a
ConfigProvider plugin type.
Drupal 8 makes it possible for modules to define their own types of plugins using the plugin API. For example, an image effect like cropping or rotating can be provided as an
A plugin of the type
ConfigProvider satisfies a given use case for providing configuration from an extension. Configuration Provider ships with plugins for core's two use cases,
Any module can meet additional use cases not covered in core by providing a new plugin. Two such examples are configuration provider plugins provided by the Config Actions Provider and Configuration Share modules.
For those interested in the technical details, here you go!
Drupal 8 modules can provide services: objects that do work and allow other code such as modules to access the functionality they provide. Services are one of the main approaches Drupal gets by incorporating elements of a framework called Symfony. For more background, see the relevant Symfony and drupal.org documentation. One type of functionality that can be exposed in a service is a storage for configuration. For example, Drupal core exposes services for the site's 'active' configuration storage and for the 'sync' storage that's used to synchronize configuration between different environments.
Drupal's configuration storage API allows configuration to be stored in multiple ways, like in the database or in files. Each configuration storage service can specify what type of storage is used. The way this is done is that Drupal core provides a storage interface. An interface is like a blueprint or contract for what a particular class (a code object) needs to do. Interfaces make it possible to have a standard answer to what a class needs to do, while leaving the question of how to do that open.
A particular class might have any number of methods, functions that are called to do something. A particular method in a class has what's called visibility, which determines where the method can be called from. Many may be private (accessible only from the class itself) or protected (accessible from the class or any of its parents or children). But the methods defined in an interface are public, meaning they can be called from anywhere. This makes sense. The whole point of an interface is to define what a class can reliably be called upon to do. So at its most basic, an interface defines a specific set of public methods a class needs to have. To implement that interface, a class just needs to have all of those methods. For more on interfaces, see the PHP documentation and a relevant section of Drupal's coding standards.
In core, there are multiple classes that implement the
StorageInterface. The active configuration service uses database storage by default, while the sync service uses files.
Okay, back to the example at hand. In Configuration Provider we don't need to permanently store the configuration, so we don't need a database or file storage. Instead, through our own
StorageInterface implementation, we provide a custom storage type:
InMemoryStorage. This does pretty much what its name suggests: hold configuration in memory. Then, using that storage type, we provide a storage service,
config_provider.storage. This is used to store configuration during the process of determining what's available on the site.
In a second service,
config_provider.collector, we provide a public method
::addInstallableConfig() that can be used to populate the
config_provider.storage storage with configuration that would be installed if all currently installed extensions (modules, themes, and the install profile) were reinstalled. Here's a sample usage:
$collector = \Drupal::service('config_provider.collector');
// Add installable configuration to the `config_provider.storage` storage.
$config = $collector->addInstallableConfig();
// Get the service for the storage that contains the installable configuration items.
$provider_storage = \Drupal::service('config_provider.storage');
// List available items.
$item_names = $provider_storage->listAll();
This approach can be used as part of an update workflow, to determine what updates are available from installed extensions. Spoiler alert: that's what's done in Configuration Synchronizer, which we won't cover until much later in this series.
Okay, confession time: to get around a thorny issue, we've resorted to what can only be called a messy workaround.
Because we're making it possible for contributed modules to define their own logic for providing configuration, we need to allow them to act when extension configuration is installed. When Drupal core is installing configuration, we need to also call our provider types. That's the domain of core's
config.installer service. So, to add our bit, we need to alter core's service.
On the plus side, the ability to tweak and refine the way a service works is built into services--in fact, that's a big part of why Symfony and by extension Drupal provides services in the first place.
When altering the default way an existing service works, there's a "right" way to do it. That's to use a pattern called "decoration"--see the relevant Symfony documentation. By decorating a service, we add just the specific customization we need, while leaving the way open for other modules to also make their own tweaks. For more on decorators, see the blog posts Drupal 8: Service Decorators and Get a decorator for your Drupal home.
But whether or not decorating a service works depends on what we were talking about before: the interface that backs a service. As noted above, an interface defines a set of public methods a class needs to have. When you're decorating a service, it's just the interface-defined public methods you've got to work with.
And there's the rub. In our case, the method we need to alter is
ConfigInstaller::getConfigToInstall()--and it's not public but protected. Again, this situation probably reflects the use case core was designed to meet. Since the assumption was that extension-provided config is only installed once, there was no need to make the
ConfigInstaller::getConfigToInstall() method public.
Since decoration is out, we need to resort to a more heavy-handed approach: a service alter.
In our alter, we swap in a custom class that extends the core
ConfigInstaller class to provide our own
::getConfigToInstall() method. In it, we pass the request to all available configuration provider plugins. We also pass configuration data provided by previous provider plugins, allowing a plugin to accomplish tasks like (a) fulfill configuration entity dependency requirements on demand (as in Configuration Share) or (b) modify configuration prior to it being installed (as in Config Actions).
Altering the service in this way is heavy-handed because it can only be done once on a given install. If another module used the same trick, one of the alters would fail, leading to breakage. Hence the appropriate warning in the drupal.org documentation:
you should use caution when making use of this feature.
Configuration Provider kinda fills the need. But are there ways it could be improved? For sure.
Configuration Provider introduces plugins that allow altering of configuration storages. That's more than a little reminiscent of what Config Filter does in a much more general way. So one option would be switch to using Config Filter plugins instead.
But Fabian Bircher, the intrepid developer of Config Filter and the related Configuration Split module, is working up a new and improved architecture for a "configuration transformer API" in this core issue. The basic idea: instead of plugins, pass and transform configuration using Drupal's event and event subscriber pattern. So, likely, whether in Drupal core or in contrib, a future iteration of Configuration Provider should wait until that proposed API is ready to roll.
Or the need could be met directly in core through a switch to an event-driven architecture when marshalling extension-provided configuration to be installed--or updated.
Related core issue
Stay tuned for the next post in this series: Configuration Snapshots.