What can Joomla learn from Drupal about distributions?

Browsing joomla.org, I recently noticed the following in the Joomla roadmap:

The Joomla! CMS [content management system] seeks to create a variety of distributions of the CMS to address a variety of common niche markets.

Good idea!

As a Drupal developer with a longtime focus on distributions ("distros"), I'd like to support this Joomla initiative. As a first step, as followup to posting in the Joomla development forum, I thought I'd try to write up some observations on Drupal, Joomla, and distros.

I've written this with an intended audience of Joomla developers. But if you're a Drupal contributor, please wade in as well!

What is a distro?

Technically, any time you package up and distribute a version of Joomla that's different from what's on joomla.org, you've got a new distribution. Great! But that's not really what I'm talking about here.

I'm talking about: how can you produce a bundled version of Joomla that provides free and out of the box pretty much everything that's needed for a particular type of website?

Say, for example, I'm going to build an ecommerce site. Rather than having to select, download, install, and configure and integrate a bunch of plugins, plus maybe write some custom code, I can instead just download and install a version of Joomla that has everything I need. Presto, there's my ecommerce site. Yes, from there I might hire someone to train me or extend the site, but out of the box I get all the basics covered.

And not just on some hosted online service. For example, the software as a service Nonprofit Soapbox Joomla-based product is sometimes, mistakenly, referred to as a distribution. Nope. For starters, it's not distributed. It's not even open source.

Some distros may focus on bundling software or solving a technical need rather than creating a preconfigured site for a particular sector. In the Drupal world, this type of distro includes Acquia Drupal, which bundles Drupal core with a bunch of contributed modules, and Pressflow, which provides performance enhancements. This type of distro serves a need. But, again, it's not my primary focus here.

Instead, I'm talking about the niche-focused distributions hosted on drupal.org. Like the Open Atrium groupware, Commerce Kickstart for ecommerce, and OpenChurch for, yes, churches. And if you're wondering who I am and why I'm interested, have a look at the Open Outreach distro, of which I'm a lead developer.

Why distros?

Given the millions of Joomla (or Drupal, or Wordpress) sites out there, there's a ton of duplication. Yes, it's great to have a CMS that's flexible, but endless repetition of variations on the same custom site development kinda defeats the purpose of open source development. Not to mention it gets tedious.

A niche-focused distro captures the configuration you'd otherwise repeat and presents it as a starting point. Further development can benefit not just one site but all adopters of the distro. Like other Joomla or Drupal development.

Some Drupalisms and their Joomla equivalents

For lack of an established vocabulary, each CMS develops its own jargon, often - just to make things extra confusing - taking the same terms and giving them different meanings. See Plugins, Modules, and Themes, Oh My! for a handy glossary.

Exportables and bundles of configuration

Before digging into specifics, some key concepts by way of background.

A niche-focused distribution is more than a bunch of general-use extensions. It's all the configuration that's the stuff of site building. And it's not just about a one-time initial setup. Sites on a distribution also need to get updates--to configuration as much as to code.

The obvious shortcut to a distribution is a variation on a database dump. That's basically the approach outlined a couple of years back on how to use a commercial backup extension to roll your own Joomla distribution. It's also the approach that was taken for the first Drupal distros. Just configure a site the way you like it, export it at that point, and you're set to clone it as many times as you like. Easy! Quick!

But not so flexible or updatable. What if some of your users don't need or want all the stuff you enabled? What if, next week, you extend your configuration or fix a problem--how do existing users get your fix?

In Drupal, the answer is so called "exportables"--configuration items that are exported into code. See an overview of exportable configuration in Drupal. While it's possible to export a whole site's configuration into a single bundle, for purposes of distro development what's preferable is to export into distinct bundles of configuration, each meeting a specific use case. That way, users of a distro can pick and choose, customizing their site with just the functionality they need. For example, a project management distro might include several optional bundles of configuration: project, group, issue, event, and so on.

In Drupal 7, the Features module is the standard tools used to package up bundles of exported configuration. As well as a UI for creating and managing "features" (configuration bundles), it provides bridging for types of configuration that don't natively support exportables.

For Drupal 8, the Configuration Management Iniative (CMI) has extended exportables to cover all types of coniguration. A recent blog post outlines how exportable features are shaping up in Drupal 8. The CMI is one of the elements slated for inclusion in Backdrop, a recent Drupal fork.

Distro enablers and what Drupal does--and doesn't do

So, in no particular order, here are some notes on what Joomla might be able to learn from Drupal. I've arranged them around a number of "distribution enablers"--that is, factors that - at least for Drupal - have been important in enabling or supporting distribution development and adoption. I've also added some notes about Joomla--sometimes just questions, since I'm a Joomla newbie. If you can answer the questions, please do and I'll update the list.

Support in core for distributions

  • Why it's important

    As a distro developer, I want the core software to work with me--not against me.

  • How it's done in Drupal

    Drupal supports natively multiple "install profiles", which are similar to modules (extensions). Install profiles can alter the site installation process, adding their own steps. They can also include plugins (modules, themes, libraries). Core includes two install profiles, "standard" and "minimal". To create a distribution, you simply add a new directory, e.g., profiles/myprofile, and put the profile for your distribution there, along with any contributed plugins it requires. See documentation on distribution development.

  • Room for improvement

    The install profiles that ship with Drupal 7 core aren't particularly useful as models, as they rely on direct database or API calls rather than exported configuration. This should change in Drupal 8, thanks to the Configuration Management Iniative (CMI).

  • Joomla notes

    Is there an API for modifying the Joomla install steps? Where is relevant documentation?

Infrastructure to support distributions

  • Why it's important

    Packaging up a distribution is a lot of work. As a distro developer, I want some tools that make the job easier.

  • How it's done in Drupal

    While the first Drupal distros were hosted externally, the drupal.org site now includes highly specialized infrastructure support for developing and packaging distributions. Developers focused on drupal.org infrastructure development added this in two rounds of community-supported development. First,  the core infrastructure was put in place. Then in a followup round of development, infrastructure devs funded by companies with an interest in distros put in place a major set of enhancements. The Drupal Association played and continues to play a key and direct role in funding, coordinating, and providing this and other Drupal project infrastructure.

  • Room for improvement

    The solutions require a lot of custom code, something that's added to the upgrade burden for drupal.org and is part of the reason the site is still on Drupal 6, years after Drupal 7's release.

  • Joomla notes

    Joomla legacy code has the JoomlaCode repository, while core has moved to Github, supported by the jIssues application. What would it take to extend jIssues to support distro packaging?

Source code and updates available through publicly accessible repositories and download packages

  • Why it's important

    Building a distro typically involves packaging up a lot of extensions along with core and supporting libraries. To use the code, it's not enough that it has an open source license. It also needs to be available.

  • How it's done in Drupal

In Drupal, developers and shops typically sell services--not code. The drupal.org site offers more than a listing of plugins. Rather, drupal.org provides a (Drupal-based) project management solution for Drupal code projects, including Git source code repositories, release management, a ticketing system, and security oversight.

  • Room for improvement

    A custom project management solution is costly in terms of development and can easily fall behind other available options. Some Drupal developers are drifting elsewhere, mostly to Github.

  • Joomla notes

    Much of the Joomla scene centres on selling software. Even presumably "non-commercial" projects often include blocks to code access, like mandatory registration on a host site.

    If there's one key barrier to distros in Joomla, I'd say this is it. Commercialized code sets up huge barriers to distro development, since as a distro developer what I need above all is the ability to package up and use the best my open source software has to offer. If that code is stuck in some private repository, it's a drain rather than a gain.

    The JoomlaCode site featured some relevant infrastructure and the newer jIssues application is promising.

    Probably the single most important step to enable distro development in Joomla would be to strengthen and promote publicly accessible code. This might require changes to the extensions directory to, at least, provide profile to extensions with publicly accessible source code--ideally, code managed through a Joomla community infrastructure including jIssues.

Strong GPL protections

  • Why it's important

    As a distro developer, I don't want to have to be concerned with licensing issues and the integrity of the code I'm including.

  • How it's done in Drupal

    Rather than relying on soft, voluntary compliance, Drupal includes clear developer documentation and direct protections for GPL integrity, including enforcement and removal from drupal.org code hosting. Longtime Drupal contributors including Gerhard Killesreiter (killes), Larry Garfield (Crell), and Derek Wright (dww) have provided focused and sustained leadership in this area. GPL protection is written into the distribution packaging system on drupal.org via a whitelist approach. Non-drupal.org code must be explicitly whitelisted before it can be packaged into a drupal.org hosted distro.

  • Joomla notes

    For someone coming from Drupal, Joomla looks in comparison (no offense intended!) like a GPL wild west. One of the first "non-commercial" extensions that I browsed from a link on the Joomla extensions listing offered a "personal use only" download and then wanted to charge for a "commercial use" version. Without a central repository with clearly enforced rules, I'd be at a loss as to how to ensure code integrity, short of a painstaking process of individually evaluating each package I wanted to include.

Update notifications

  • Why it's important

    As a distro developer, I include a lot of code in my distro and I need an easy way to make sure it's all up to date.

    As a distro user, I want to make sure the code I'm running is up to date, particularly for any security releases.

  • How it's done in Drupal

    The Drupal update manager allows sites to poll a central server that provides update data in an XML format. Since almost all Drupal code is hosted on drupal.org, projects get update notifications out of the box.

  • Joomla notes

    Joomla core includes update notifications. For extensions, update notifications rely on the individual extension authors implementing an update server. In practice, only a subset of extensions support updates.

Strong institutional backing for distro development

  • Why it's important

    It's a lot of work to build out and support the enabling toolset for distributions and the distros themselves.

  • How it's done in Drupal

    The Drupal distro toolset was built out alongside the first "modern" Drupal distro, OpenAtrium. Development Seed, a (then) Drupal shop, built OpenAtrium as a generic tool for use by large clients including the World Bank, and at the same time built out the Features toolset. When Development Seed changed its focus, OpenAtrium and the enabling toolsets moved to Phase2, a company already focused on building out Features-based distros including OpenPublish. Most of the well maintained distros are produced by Drupal shops that generate business from building sites with their distros. In many cases, distro development includes a range of other independent contractors or shops that also build sites with the distros and contribute plugins and other enhancements.

Standard site-building tools

  • Why it's important

    As a distro developer, I need tools to build out configuration, and the more standardized those are the better.

  • How it's done in Drupal

    In recent major releases, there's been an increasing move to bring key configuration management tools into Drupal core, including custom content types (added in Drupal 6), fields (added in Drupal 7), and Views (added in Drupal 8). Even in contrib, there is a tendency for usage to converge on a best solution. With Views, for example, there wasn't a question of which query builder would be a candidate for core inclusion--Views was already the clear choice. Factors that facilitate this convergence include (a) non-commercial code, so that the best solutions are universally accessible and (b) the usage statistics ranking of projects on drupal.org, so that site builders easily see which solutions are gaining adopters.

  • Joomla notes

    Joomla looks to be relatively challenged in both standard tools and the bringing of those tools into core. Consequently, it may be a bit harder to build integrated solutions.

    Standardizing on site building solutions could be a key step.

Extensions that facilitate distribution development

  • Why it's important

    As a distro developer, I want to have handy tools I can use both to author and to deliiver my distro.

  • How it's done in Drupal

    Drupal contributors have produced a standard toolset for producing Drupal distributions. At the core is the Features module, which includes an advanced UI for capturing and exporting bundles of configuration that meet a particular use case.

  • Joomla notes

    Are there equivalent extensions in Joomla?

Command line tool and packaging format

  • Why it's important

    Distro development involves a bunch of repeated tasks, like exporting bits of configuration. Please, save me endless clicks! It also involves gathering up code from disparate places.

  • How it's done in Drupal

    Alongside Features, the Drush command line tool includes key functionality supporting distro development. The "drush make" format makes it possible to specify, in a simple text file, a custom set of code that will be packaged up to produce a full distro.

  • Joomla Notes

Is the Joomla CLI an equivalent base here?

Flexible core

  • Why it's important

Key to being able to produce radically different distros on the same code base is ensuring that very little in the core CMS is fixed or required.

  • How it's done in Drupal

    Drupal 7 has a minimum of required core modules. Modules providing more specialized functionality have steadily been moved out of core into the contributions repository.

  • Joomla notes

    There have been efforts to build a "light" version of Joomla to facilitate distributions, including Square One and JLite. These are useful examples, but they've hit the same barriers that early Drupal distros did--it's way too much work to have to maintain a largely forked version. Instead, it will be important to focus effort on refactoring the core CMS so it has a minimum of required components.

    Tangentially related: there was public musing about whether Molajo was a Joomla distro or a fork, but according to lead developer Amy Stephens it's neither--see comment below.

Distribution-friendly hosting solutions

  • Why it's important

    As a distribution user, I want hosting options that make it easy to install and update my preferred distro.

  • How it's done in Drupal

    Many Drupal hosts offer special support for distros. The Drupal Aegir hosting platform includes built in support for distributions. The Pantheon system supports distributions.

  • Joomla notes

    Are there Joomla hosts interested in supporting distros?

Further reading on distros in Drupal

Key Joomla strengths

Looked at from outside, Joomla has some tremendous strengths that should provide a solid base for building out distro support.

  • Huge install base

    The sheer number of Joomla sites out there means there's a lot of potential market for distro-based sites. As site owners face the inevitable need to upgrade to new versions, getting off a customized one-off site build and onto a distro can be a big attraction. Upgrading an existing site to a distribution can save a ton of time and expense in upgrade costs and future maintenance.

  • Strong production and community leadership

    Having an inclusive and participatory structure should help a lot in marshalling resources to achieve the (significant, but doable) tasks of enabling and supporting distro development.

  • Strong developer and adopter communities

    There's no shortage of skilled Joomla talent to draw on.

  • Institutional backing

    It takes a lot of backing to successfully make changes and fund infrastructural development. Open Source Matters and other key Joomla institutions would be important players.

Next steps

So that's it for an initial brainstorm. To emphasize again: my aim here is to start a conversation about what Joomla devs might learn from the Drupal experience re dos and don'ts of facilitating distro development. I'm not at all suggesting that Joomla should follow Drupal's "model"--just that a close study of how Drupal has done distros is probably a valuable input into making it happen in Joomla.

What did I get wrong? What did I miss? Where can we take the conversation from here?

Feed: 

Comments

Good resource from DrupalCon

Good resource from DrupalCon Prague by Jeff Eaton talking about distributions with Drupal 8.

I continue to work on Molajo. It's not a distribution of Joomla. It's not a fork of Joomla. It's brand new code. I've offered several packages to the platform team but to date, there has not been a lot of interest. In their defense, the framework team is trying to repackage the core Joomla platform as packages and maybe that's enough for now. Still open for sharing. =)

Getting close. It's been quite an adventure, made a lot of mistakes a long the way, learned a lot, and feel pretty good about the codebase at this point. Looking forward to rolling it out when it's ready. It'll be a great tool for building distros. Here's an example of what it takes to build the equivalent of a Joomla Component. Molajo Articles are like com_content with tags and comments. This code is written in the same way that Drupal 8 is and I hope to be able to integrate into that application, as well.

Best.

Hi Nedjo

Hi Nedjo

I really appreciate such a thoughtful and knowledgeable post, even if you call one of my old blog posts "mistaken" early in the piece :)

Something interesting came up from the Commerce Guys the other day: 

"Only 1.2% of Drupal sites are distributions and 60% of those are Drupal Commerce"

So all of those other distributions produce about 0.5% of all Drupal sites.

That's a lot of work by a lot of devs to reach a small audience. It is worth it?

Good question

Yes, the apparently low adoption of Drupal distros is important to look at closely.

Mitigating factors include:

  • Sites based on distributions may be less likely than other types of sites to enable update notifications, without which they won't show up in statistics.
  • There have been report issues with stats gathering of distro-based sites.
  • Some of the higher profile distributions like Open Atrium and Drupal Commons were hosted off of drupal.org for Drupal 6 and so don't show up in statistics and were ported relatively recently to Drupal 7.

But these factors are only a small part of the picture. However we look at it, distro-based sites are a small fraction of Drupal sites. The reasons are not clear, but here are some of my hunches.

  • There is if anything an embarrassment of riches in Drupal distros. The barriers to entry are almost nonexistent--package something up and post it and you're a distro author. Consequently, there are hundreds of distros, varying greatly in quality and kind.
  • Distros are all or nothing. With modules or themes, you can easily try one out on a site and, if you don't like it you just uninstall it. But with distros, if you don't like it, you have to start again from scratch.
  • With a few exceptions, distro producers have struggled to find a viable financial model. Maintaining a distro is a ton of work. Many distros were produced as one-offs - for example, as part of a funded client project - but then languished when funding dried up.
  • Drupal 7 lacks much standard core CMS functionality (like WYSIWYG editing) and so distro authors/maintainers are burdened with developing and maintaining their own solutions.
  • Organizations looking for something that "just works" may look first to Wordpress.
  • Drupal distributions may have a low visibility outside the Drupal community.
  • The mainstay of Drupal shops and contractors remains custom site development. Shops and developers see less to gain by building off of what they perceive as someone else's solution. See also my previous blog post Drupal distros sound great, but....
  • Despite efforts at promoting interoperability, distros remain for the most part mutually incompatible. This means that (a) each distro must maintain a whole slew of features that overlap with those of many other distros and (b) users of one distro are cut off from anything produced for a different distro.

Thanks Nedjo

Thanks Nedjo

You're enormously informed on this and appreciate you taking time to respond in detail.

I don't have a whole lot to add that wasn't in my original 2010 post, except that I've started to learn more to towards the "con" side, as you have.

Distros generally seem to fall between too stools, not fully meeting the needs of beginners or experts.

 

I think the best place to

I think the best place to continue the conversation is at github.com/joomla-cms/start-here - that's the home of the distributions project (but there is far, far more depth to the project than just being able to create a "Joomla Lite" - my hope it becomes a complete paradigm, if not a cultural shift).