Karma in Launchpad

For some time, I have been curious about the karma system in Launchpad – the massive infrastructure being erected not only for Ubuntu, but for the F/OSS community in general.

The karma system gives “karma points” for actions which the system appreciates, like bug reporting, specification writing, bug fixing and so on. But so far, it does little with this information apart from showing who the top 5 contributors in all of Launchpad are (lower left hand corner), and show how the karma is divided up between various tasks.

Now, with so much else around Ubuntu and Launchpad, it didn’t appear out of nowhere. The karma system is obviously inspired by systems in other F/OSS communities. And I thought it would be interesting to examine how these compare to Launchpad. The two that I’ve noted (though there are undoubtedly more) is Slashdot and the GNOME Bugzilla.

In both Slashdot and GNOME, karma or bugzilla points play an important social function. It is a value of trust.

Slashdot
Slashdot karma is gained by writing insightful or relevant comments or posting interesting new stories which are moderated as such by the other readers. In turn, you can use (expend) this karma to moderate other posts as insightful or not, thus strengthening the system. If you’ve been modded insightful with +5 modifier, chances are that you will also apply that good judgement when moderating other posts, so the system gives you 5 points to mod posts where you see fit. Thus the system trusts you the more you contribute. On Slashdot, karma doesn’t have an easily accessible point value but rather vague textual identifiers such as “Terrible”, “Neutral” and “Excellent” – this is to avoid some people gathering huge amounts of karma and thus setting themselves above moderation. The Slashdot FAQ has this gentle reminder on the topic:

Karma is used to remove risky users from the moderator pool, and to assign a bonus point to users who have contributed positively to Slashdot in the past. It is not your IQ, dick length/cup size, value as a human being, or a score in a video game. It does not determine your worth as a Slashdot reader. It does not cure cancer or grant you a seat on the secret spaceship that will be traveling to Mars when the Krulls return to destroy the planet in 2012. Karma fluctuates dramatically as users post, moderate, and meta-moderate. Don’t let it bother you. It’s just a number in the database.

The GNOME Bugzilla
The GNOME Bugzilla on other hand, works somewhat differently. The stated goal of the point system is to make make each contributor’s level of experience with Bugzilla and GNOME clear in an easily accessible manner. On a typical bug report, you will see the reporter’s point score prominently displayed, and when people comment, their point score will be displayed as well.

At the GUADEC, I talked to Olav, the GNOME bugmaster and maintainer of the GNOME Bugzilla, about how the system works and what they use it for. I asked whether the point system wouldn’t encourage long-time contributors to ignore bug reports from newbies, but he explained, that quite to the contrary, these long-time contributors could spend more time on newbie bugs, because they could rely on the bug information from other people with high point-scores and thus didn’t have to triage these as thorughly. It also helps new users to see which comments are from experienced people whose advice they would do well to heed.

The Bugzilla point score is calculated from the following formula:

log_10(1 + #comments) + log_2(1 + #bugs_closed) + log_2(1 + #bugs_reported)

The point scale is logarithmic with different bases for comments (log10) and bugs closed and bugs reported (log2). That means that you need to make 10 comments to get a point, but only close 2 bugs to get another. But for the next point after that, you’ll need to reach 100 comments, or close 4 bugs. Then 8, 16, 32 and so on. This means that it gets progressively harder to gain more points as you go along.

This means that going from 0 to 1 is very easy, but already at 5 points you can be considered a proficient bug hunter. Reaching the mid-20s range like some of the core developers will require a much greater effort. So far, only one bug hunter has reached 27 points. A list of how the points are distributed among the registered Bugzilla users is quite telling.

This logarithmic increase in points almost entirely cancels out the “computer game” tendency addressed on Slashdot, but the points themselves do not easily indicate a level of expertise to newbies. 27 points doesn’t sound like much compared to Launchpad’s several hundreds of thousands of points, but when you look at the statistics behind it, it is somewhat more impressive:

Since all bug management actions are counted towards the same score, you can access the bug management information, and can thus directly see what you have done to make you point score increase by looking at the list of bugs reported/closed/commented on the contributor page.

The GNOME Bugzilla aptly uses these statistics to generate weekly reports which give an idea of the amount of work being done within the last week. There are also annual statistics list which is announced and examined with much interest among the GNOME bugsquad.

So in this way, there a two distinct set of “point values”. There’s an overall contribution to GNOME value, and there is a contribution within a set period of time (last 7 days, last calendar year). This makes it possible to represent and acknowledge the experience and contributions from each individual both in total and over a shorter period of time.

In order to solve the problem of the point score giving a level of trust that is not easily interpretable for new users, the GNOME developers have introduced that developers can register the packages they maintain so that this information will also appear on the bug report, like this:

There has also been a discussion about turning the point values into some easy-to-understand text alternative. I would imagine that they could well end up with something silly and similar to the titles of the individual class levels in Dungeons & Dragons (“congratulations, you have reached level 23 – you are now a Grandmaster of Flowers!”). One GNOME developer has even compared developing GNOME to playing online role playing games!

Launchpad
Since Launchpad is much more than just a news forum or a bug tracker, it’s Karma system needs to span over several different elements that are not easily alignable. Launchpad consists of:

Malone – the bugtracker
Soyuz – the distribution management software
Blueprint – the Specification Tracker
Rosetta – the translation software
Bazaar – the version control system
– the so-far unnamed support tracker

Each of these distinctive parts have different uses and thus offer different ways of gaining karma. Different actions add different points to your karma total. For instance, reporting a new bug gives you 10 points, while providing an (unaccepted answer) on a support request gives you 1 point.

The developers admit that they’re just guessing when it comes to what right point values for each action should be:

The score associated with each action type is currently just a best guess. We have yet to do any serious analysis on the real data to determine which actions should be rewarded more and which actions rewarded less.

The way the system is built makes it possible to award karma points for actions which aren’t related to any of the above parts of Launchpad, such as registering an email address or a GPG key. This may further dilute the relevant data from the different parts of Launchpad.

All of a user’s karma points from all over Launchpad are then calculated with a degradation algorithm, where contributions today are worth 100%, contributions 182 days ago are worth 50% and contributions made 365 are worth 0% – which means that if you don’t contribute for a year, you karma will drop to zero. The developers provide the following rationale:

We feel it important that Karma is reduced by time, as otherwise new users may feel that they have no hope of catching up with a long time user, even if that long time user is no longer active.

This means that the Launchpad karma cannot be used to measure longtime experience or social trust within Launchpad, as trust or ability does not degrade like this. Instead, Launchpad karma is solely a measure of activity which is recalculated on a daily basis. This means that the karma values can fluctuate wildly from day to day – especially depending on how certain tasks are rated. For a long period of time, the Rosetta developers had way too much karma compared to their work because Rosetta gave out karma points more easily than eg. Malone.

The use of karma in Launchpad has been discussed on the Launchpad-users mailing-list, and some bug reports have been filed on it to base the karma on logarithmic factors like the GNOME Bugzilla and displaying the karma next to the contributor’s name on a bug report. Also, there has been bug reports asking for an easy way to use the list of karma gained as a log through which you can access the comments and bug reports that you wrote to get the karma in the first place.

And yet another bug report has been filed in the hope that more statistics on karma distribution will be available than the current top 5.

Being part of a immensely complicated (and unfinished) system, the Launchpad karma system has a lot unresolved issues. First of all, it doesn’t even seem like the Karma system has a solid purpose yet. From the specs it seems like it was mostly to log user activity. But since the user can’t use the log to backtrack to the actions they performed, it is not that helpful in that regard. In the Malone, there is talk of using Karma as an incentive for people do good work. But if they have trouble figuring out how the karma is counted, and if the karma drops back to zero after a year, then that is not a very good incentive.

The central focus of the Karma system should not simply be activity, but trust. Launchpad will used by a lot of people, and their interactions are based on very few factors. Having social guide line such as Karma can help a lot. Both Slashdot and the GNOME Bugzilla provide examples of how that can be done successfully.

I’m afraid that the Launchpad developers have been too caught up in the technicalities to think of the actual use cases of the Karma system. I hope this will help a bit (I have also sent a link to this text to the Launchpad-users mailing-list).