Debian as the research library of Free Software

I’ve had a quite interesting discussion with Lars Risan in the comments to my recent blog post on Launchpad about Lars’ paper on the role of technical infrastructure in Free Software development, and I think Lars does well to describe the central tension within these:

I think Debian is a tremendously interesting case when it comes to understand human cooperation. Because, not unlike what Eric Raymond pointed out long ago, there is the “Bazaar” of Debian, the heterogeneity of the network. And there is the “Cathedral” of it: The ability to structure the large amount of work to the degree that you can slip a Debian DVD into a Windows-computer, and turn it into a Debian computer in less than an hour. How far can you take this mix of “cathedral” and “bazaar”? How much bazaar can you have before it forks, and how much central control can or must you enforce upon the network? How can you build a system that enables the “beauty of (hacking) mankind” to simply do good, and/or how much must you invoke something like the (partly fantasy figures) of the Ubuntu-Launchpad and the Debian-Cabal?

This much-discussed question of centralization connects well with the stated goal of World Domination that many Free Software communities state again and again in a typical “ha-ha, only serious” fashion.

Global domination is a centralistic quest to compete with the cathedrals of Windows and Mac OS, and to do this while maintaining the freedom of learning and technical exploration that is the essence of Free Software will inevitably be a balance act. It is one that all Linux distributions are performing with various degrees of success.

One way is, as Canonical proposes, to let companies define the supported Free Software that the end user might need, and guarantee that it is available and that it “just works”. Thus creating small pools of centralization – the specific traits of Ubuntu – merging with the larger decentralized pool of shared knowledge within the bazaar – Debian Unstable every six months.

Mark Shuttleworth describes this relationship as Debian as “the Tibetan Plateau of the free software landscape” upon which Ubuntu is built:

By contrast with Debian’s Plateau, Ubuntu is a cluster of peaks. By narrowing the focus and allowing the KDE, Gnome and server communities to leverage the base of Debian without treading on one another’s toes, we can create a K2, and a Kangchenjunga and a Lhotse. Ubuntu’s peaks depend on the plateau for their initial start, and their strong base. Ubuntu needs to be humble about its achievements, because much of its elevation comes from Debian.
[...]
Many people have asked why I decided to build Ubuntu alongside, or on top of, Debian, rather than trying to get Debian to turn into a peak in its own right. The reason is simple – I believe that Debian’s breadth is too precious to compromise just because one person with resources cares a lot about a few specific use cases. We should not narrow the scope of Debian.

At last year’s DebConf, a talk compared Ubuntu’s use of Debian to shopping at a supermarket, a one-stop shopping for your free software needs. This innocent analogy sparked a lot of discussion, with many of the Debianistas arguing that this reduced Debian to a giant collection of un-integrated software – which wouldn’t be very interesting at all. As Debian Developer Joey Hess put it:

My main motive for contributing to Debian is to make Debian the best distro I can; I don’t mind if others use that work, especially if stuff gets contributed back. But it’s long been clear to me that the most important added value to Debian is not adding another package to the shelf, but finding new ways to integrate our software together. When you’re working mostly above the level of individual software packages, to have your work mostly appreciated on the basis of “component contained in Ubuntu” is not very motivating.

Integrating and creating a universal distribution is still a central motivation for many Debian developers, who may find that Ubuntu is not only stealing their thunder but also compromising their ideals, again with no small amount of distrust towards the dealings of the opaque corporate entity that is Canonical.

But as Shuttleworth also notes, that when you seek narrow goals, it will come at the cost of openness and democracy. The Debian community is obviously unwilling to make such a trade, and as such, must look elsewhere for inspiration on how to organize themselves towards their goal.

Taking a cue from an earlier comparison of the Free Software communities to the scientific communities, I think it would be worthwhile for Debian to leverage their solid technical infrastructure towards becoming commonly agreed upon open standards to allow the exchange of knowledge and code will be possible on equal terms. In that way, I think that Debian is quite well poised to become a sort of “research library of Free Software” – collecting the “monographs” and “articles” of code, cataloguing and organizing them for easy and open access.

Almost all Debian developers, and apparently 76 % of all Debian users,use Debian Unstable or Testing.

Debian Unstable, codenamed Sid, is the latest, most volatile version of Debian, where new packages are uploaded to, and changes to old packages are added on a daily basis. It contains all of the latest releases of the upstream communities, packaged and categorised. Debian Testing contains all of the packages that haven’t had a critical bug filed against them after spending 10 days in Unstable.

Only 24% of the Debian users use Debian Stable, since the releases are so rare as to only come out every 2 or 3 years – much too rarely to keep up with all the latest hardware and desktop applications, making Debian Stable relatively unattractive in the long term.

Since Debian Unstable (or Testing, for the less adventurous) fulfills the needs for both the users and the Debian developers, there is little incentive to make an Upstream Version Freeze and actually get a new release out, and since Debian is solely based on volunteer work (with suggestions to pay release managers meeting fierce opposition), bug fixing and release deadlines aren’t enforced that well, further slowing the actually release of the Debian Stable.

As Shuttleworth points out, Debian Unstable is “not subject to the same political tradeoffs that are inevitable when dealing with releases, architectures, dates, deliverables, translation, documentation and so on.” – it is the core strength of Debian.

As the developers I interviewed said again and again when asked what they get out of contributing to Free Software projects like Debian: “I get a system that works for me.” To some extent I think that is what Joey Hess’ means when he says that his main motivation for contributing is “to make Debian the best distro I can.”

Debian Unstable is not merely about keeping software up to date, but also about improving it, further integrating it within the Debian infrastructure. It is a research library in the way it is not only focused on gathering knowledge, but also on using it. It is also a laboratory where Free Software is shared and combined in new and interesting ways.

Maybe that would the most fitting analogy to what Debian really is: A research library containing all of the Free Software that lives up to set strict criteria. Offering all the development tools, libraries and applications up-to-date for new experimentation.

This would fit well with the Lars’ counter-argument:

Science is (also) extremely disunified (there are 22.000 different medical journals in the world. “Medicine” (in singular) is known by no-one). And “science” can live with this relative disunity. Can Debian? There is at least one difference: “Debian” as a whole must occasionally come together to produce a release. A release which is unified enough for a computer to work. Fuzziness is simulated by computers, because they work only by strictly separated ones and zeroes. The knowledge of Debian, packed together as a release is pretty much a unity. A unity of which a large degree of coherence is required. The knowledge of science may be unified in an encyclopaedia, but, metaphorically, the discrepancies and glitches of that body of knowledge is so enormous, so “buggy”, that it would make any computer halt very early in the boot.

Much like the knowledge of science only can gathered for release in an encyclopaedia that is outdated before it is even published, all of the software within Debian will also be outdated by the time it is released. Perhaps Debian should take inspiration from Wikipedia rather than other Linux distributions.

Wikipedia is a reservoir of such diverse knowledge that it would never be in a position where it could be published in paper form. It would require an insane amount of work, flamewars and unhappiness for all involved. It would require a centralization that simply is not present in the Wikipedia community.

But in spite of this, by far the biggest part of Wikipedia is eminently usable and accessible at any given time. As is Debian Unstable.

Perhaps the Debian community should take some comfort in this.