Weaknesses of non-dictatorial community software projects

As an inveterate user and observer of open source, I've read many criticisms of open source software development that simply don't hold water. This post is about one criticism that I believe does hold water.

In contrary to the straw man often erected by anti-open source pundits, most open source projects have a relatively formal management structure wherein one person or a committee exercises strong top-down control over the project's canonical source code repository. That dictatorial hand may be felt lightly or heavily, but at the end of the day decisions must be made, and priorities set, and the project will follow the dictator/committee's decisions.

(Or --- in the case of a fork --- not, but nobody likes forks and ultimately each half of the fork generally assembles some sort of management structure.)

However, not all open source projects have a dictator. A few projects manage to survive with only informal technical governance structures. The largest and most successful of these is probably KDE, a desktop environment. KDE has a number of associated formal legal entities responsible for things like legal and marketing issues, but the technical direction of the project is not managed by either a benevolent dictator or a committee.

There are good things and bad things about this. Here is one of the bad things: this KMail wishlist item has been in the bugs database since December 2000. It is the top vote-getter on the most requested features list. But nobody has tackled it. A couple of people have made gestures in the direction of implementing it, but no bug-closing patch has been committed to the trunk.

In an open source project with a dictatorial or committee-led governance structure, somebody would long ago have cracked some heads and gotten this feature implemented. In a commercial software project, open source or non-, some engineer would be assigned ownership of this feature; and goddammit, if that feature didn't get implemented and maintained, that engineer would be fired and the feature would be assigned to someone else. But KDE's headless. It's less like a mammal with a central nervous system than an enormous amoeba whose various pseudopodia ooze tropically in the direction of "developer itches" and "coolest implementation hacks" (hence the recent proliferation of "hugely ambitious infrastructure refactoring" subprojects like Plasma or Solid) rather than unsexy, annoying-to-implement features that merely provide value to end users.

Now, as a by-product of this process, you often end up with a lot of excellent software. I use KDE every single day, and on balance I prefer its interface to both Windows and Mac OS X for intensive use.[0] But this software process isn't rationally optimized to serve the end-user, and so there will always be these frustrating little blind spots.

Which isn't to say that commercial software processes are rationally optimized to serve the end user either, but that's another whole series of posts waiting to be written.

(See also: Bug 8333, Bug 55777.)

[0] Yes, that's right --- not the code, not the customizability, not the Free and open source license, but the user interface of KDE is preferable, for a heavy user, to either of the big consumer operating systems. Windows XP and Mac OS X ---especially the latter --- are both highly polished and acceptable for casual use. However, when I'm juggling several terminals, eight or nine Emacs windows, and a half-dozen Firefox windows, the extra features of KDE's window manager simply destroy even Mac OS X's. Ironically, some of KWin's best features were in the classic Mac UI, and were dropped in OS X. (I first got hooked on windowshading when I was an undergrad Silicon Alley web production intern, juggling BBEdit, Photoshop, and multiple versions of Netscape. Ah, the old days...) Other features, like window gravity, have been pretty standard in Unix window managers for years, but have somehow never cracked consumer OSes even though they seem like no-brainers.

"Do you want America to win in Iraq?"

Apparently this is the latest sound-bite for Republicans trying to defend the Bush Administration --- see this Oct. 27 Lynne Cheney interview...

Well, right, but what is CNN doing running terrorist tapes of terrorists shooting Americans? I mean, I saw Duncan Hunter ask you a very good question and you didn't answer it. Do you want us to win?

...and O'Reilly in this Oct. 27 Letterman interview...

O'Reilly: But they don't want to hear about the bad world that we live in. It's an evil world that we live in. Let me ask you something. And this is a serious question. Do you want the United States to win in Iraq?

...and O'Reilly again on the The View Oct. 18...

O'REILLY: Hold it, hold it, hold it. Want America to win in Iraq, by the way?

O'DONNELL: I don't think it's possible.

O'REILLY: Do you want, do you --

O'DONNELL: I think it's an ill-thought-out plan and I think we should get out of that situation before Americans are killed. Out. Out of Iraq.

O'REILLY: Do you want America to win in Iraq?

JOY BEHAR (co-host of The View): What does it mean to win?

O'DONNELL: I want America to be what the founding fathers wanted it to be, a democracy, where we the people --

O'REILLY: OK. So you don't want America to win in Iraq.

The clarion call has been sounded, the marching orders have been given, and all the little troopers across America are no doubt repeating this mindless sound bite --- "Do you want America to win?" --- even as we speak.

The proper sound-bite response to this sound bite is: Yes, and I want the Mets to win the World Series this year too. But that train has left the station, and we were not on board.

Bonus additional sound bite, in case you don't like that one: Yes, I want America to win. I also like puppies and apple pie. But none of those things has anything to do with continuing the war in Iraq.

The United States has already lost in Iraq, in the only sense that matters: we have failed disastrously in every single one of our war aims. The Bush administration's holding out because it cannot admit failure, and it knows that if it delays long enough then the next administration will come in and the failure can then be blamed on them. As long as America does not withdraw from Iraq under his watch, the President can continue to indulge in the delusion that history will judge him kindly. Or, in other words, the United States continues to spend untold blood and treasure, more or less, to protect George W. Bush's frail self-esteem. A noble cause, to be sure, but one wonders if the effort is proportionate to the results.

To the Critical Asses who interrupted my commute today:

My shuttle home today was halted, idling in traffic and burning fuel needlessly, for over 20 minutes today because of you self-righteous assholes. Congratulations, fucktards, you just made an enemy!

Streets are mixed-use facilities. In normal conditions, cars, bikes, pedestrians, rollerbladers, Segway riders, and all kinds of other uses share the public space. Critical Mass doesn't inspire people to respect the streets; it comes across as a bovine herd out to spitefully annoy others. The moral equivalent of Critical Mass would be a bunch of car drivers parking on the sidewalk to make a point about obnoxious pedestrians. Do you think such an action would cause pedestrians to reform their behavior? What makes you think your gesture works any better?

And I love how this movement's so big in San Francisco. Because there's such powerful opposition to mass transit, renewable and alternative fuel sources, and liberal do-goodery in general here. That point really needs to be made in the most confrontational way in this district, doesn't it. Why don't you take your shit to Los Angeles, or better yet to Bumfuck, Utah or someplace where everybody drives all the time and votes Republican? Oh, that's right --- your stunt only works here because San Francisco is already really dense, and therefore energy-efficient.

So, I got off the shuttle and actually laid myself down in the path of the bicyclists at Larkin and Golden Gate... for about thirty seconds, before I concluded that nobody else was going to join me, and therefore my gesture was futile. Resigned to this folly, I stood up and walked home.

Although, now that I think about it, maybe I should have stayed lying on the street longer. The moral status of a pedestrian one-ups that of a bicyclist, and if I stayed long enough then maybe other people would have gotten out of their cars and laid down too.

Maybe. Next year.

I suspect this post will make me some enemies. Well, it's been a while since I wrote something flat-out obnoxious. Back to form once again.

Most effective afternoon anti-sleep trick

Chew gum.

I never used to chew gum, but caffeine doesn't work for me with any consistency anymore. Sometimes it works, but other times I can drink enough to make myself jittery and yet fall asleep in those dreadful post-lunch, midafternoon doldrums. I have never yet fallen asleep while my jaw's moving though, whether during meetings, talks, or coding at my desk.

The only major downside I've found so far is that if I chew for too long, my gums start to feel itchy, like I need to chew something harder.

This is probably the most useful thing that I will ever post here.

Incidentally, if you do this, please try to chew with your mouth closed. I hate, hate, hate the sound of human beings chewing with their mouths open, and I would not want to inflict it on anyone.

Scroll bindings (Wednesday Emacs blogging)

Part of the Zen of Emacs is the fact that --- in contrast to merely mortal text editors or word processors --- you need not move your hands from the default typing position for common navigation tasks, like moving to the beginning of a line (C-a) or word (M-b).

Clearly, therefore, it is uncivilized to use arrow keys, or to move your hand to the mouse and hunt for that little arrow on your scroll bar, when you merely want to scroll up or down a couple of lines. Accordingly, the following elisp binds downwards and upwards scrolling to the M-n and M-p keystrokes respectively.

; Handy incremental scrolling keys
(define-key global-map "\M-n" (lambda () (interactive) (scroll-up 1)))
(define-key global-map "\M-p" (lambda () (interactive) (scroll-down 1)))

I chose M-n and M-p to be mnemonic "cognates" with C-n and C-p. Note that scroll-up is so named because it moves the document up, which makes the viewport appear to scroll down, and vice versa for scroll-down.

Nevertheless, it is sometimes convenient to do scrolling with your mouse --- e.g., if your hand's already there because you're switching focus among open windows. Recent Emacsen understand the scroll wheel, but some older versions do not. Fortunately, they can be taught, using the following recipe suitable for either (FSF) .emacs or (XEmacs) .xemacs/init.el:

; Mouse wheel: scroll up/down; control-wheel for pgup/pgdn.
(defun wheel-scroll-up   ()   (lambda () (interactive) (scroll-up 2)))
(defun wheel-scroll-down ()   (lambda () (interactive) (scroll-down 2)))
(defun wheel-scroll-pgup ()   (lambda () (interactive) (scroll-up 20)))
(defun wheel-scroll-pgdown () (lambda () (interactive) (scroll-down 20)))
(cond
 ((string-match "XEmacs" emacs-version)
  (progn
    (define-key global-map 'button5 (wheel-scroll-up))
    (define-key global-map 'button4 (wheel-scroll-down))
    (define-key global-map '(control button5) (wheel-scroll-pgup))
    (define-key global-map '(control button4) (wheel-scroll-pgdown))))
 (t ; FSF Emacs uses weird [bracket] keymap specifiers.
  (progn
    (define-key global-map [mouse-5] (wheel-scroll-up))
    (define-key global-map [mouse-4] (wheel-scroll-down))
    (define-key global-map [C-mouse-5] (wheel-scroll-pgup))
    (define-key global-map [C-mouse-4] (wheel-scroll-pgdown)))))

This recipe also binds C-mouse-4 and C-mouse-5 to page-down and page-up, for handy fast scrolling, which does not come standard even on recent Emacsen. It should be pretty obvious how to customize it to scroll more or fewer lines at a time.

Friday cat blogging

Warning: ...ah, hell, if you can't guess from the title, your time deserves to be wasted.

A nervous but resilient feline. The SF SPCA got him from San Francisco City Animal Care & Control; who knows where he was before that, although his behavior indicates that he once had an attentive human home. The SPCA's veterinary records state that upon arriving at the SPCA, he suffered from extreme fear-related anorexia, leading to liver disease, and he had to be placed on a feeding tube. A few weeks later he was up and about again, fully recovered. When I found him at Maddies Adoption Center a couple of months later, Manny climbed onto my lap.

Peace never lasts. My nefarious plan for feline adoption necessarily entailed having an SPCA volunteer pack his yowling, hissing, miserable ass into a cardboard box. Evidently not one to take mistreatment lying down, he clawed his way out of the box during the ride home.

He spent the rest of the ride climbing around the car in panic, hyperventilating and hissing. At this point, Bay Area readers should imagine the drive from the Inner Mission to just north of Japantown, and the jostling it implies.

Cat adoption guidelines tell you to have a quiet, isolated room prepared with cat necessities. They're not kidding around. I set him down in the bathroom, and for the rest of the day he stayed there, retreating under the sink or behind the toilet whenever the door opened. My chief bonding tactic was to lie down on the bathroom floor and take naps. Sleep evidently rendered me sufficiently harmless that he was willing to come over and sniff at me periodically. (Maybe I should try this with women.)

Anyway, eliding some details, a dozen days later and he's out and about, accepting my food and affection. Unexpected noises or fast movements continue to make him flighty. Still, the mongrel cat has come home.

Rotating background colors (Wednesday Emacs blogging)

Like every veteran Emacs user, I've long used multiple buffers, multiple windows in a buffer (C-x 2) and multiple frames (C-x 5 2) to edit multiple files simultaneously during heavy sessions of programming or writing. Lately I've been working on a monitor so vast, and reading so many different files simultaneously, that I've found it useful to run multiple instances of Emacs as well. (Among other things, this speeds up tab-completion when switching buffers with C-x b, because each instance has shorter buffer list.).

When running multiple Emacs instances, it gets hard to tell which frame corresponds to which running instance of Emacs. The following Elisp code enables rotation among a set of background colors on a keystroke:

(defvar background-color-rotation
  '("aliceblue" "thistle" "lemonchiffon" "khaki" "papayawhip"
    "honeydew" "mistyrose" "paleturquoise")
  "List of background color names to rotate")
(defun next-background-color ()
  "Rotates among colors in background-color-rotation."
  (set-variable 'background-color-rotation
                (append (cdr background-color-rotation)
                        (list (car background-color-rotation))))
  (car background-color-rotation))
(define-key global-map [f10]
  (lambda ()
    (interactive)
    (set-face-background 'default (next-background-color))))

Paste this into your startup file for recent FSF Emacs (.emacs) or XEmacs (.xemacs/init.el). As always you can also try it out quickly by pasting into your *scratch* buffer and using C-j to evaluate the defvar, the defun, and define-key expressions in turn. Once the above has been evaluated, hit F10 to rotate among the named background colors.

The code's pretty trivial, so if you know Elisp --- or even if you don't --- it should be relatively straightforward to customize it for different colors, or a different keystroke. To list all the color names your display supports, use M-x list-colors-display.

I considered making the macro save the background color rotation between Emacs invocations, so that each new instance would automatically come up with a different background color. It wouldn't be too hard to do this, but I decided it didn't fit the way I work. I like to have a particular color correspond to a particular task --- "aliceblue" for general hacking (the first Emacs I open when I login), "thistle" for writing notes to myself, etc. --- so I prefer to do the rotation manually.

Incidentally, note the use of string arguments to defvar and defun to document the function and variable. Some Python advocates tout Python's "innovative" and "unique" use of string literals in class and function bodies for documentation comments. I agree that it's a clever idea, but it predates Python by several decades. There's a reason that Emacs is called the "extensible, self-documenting display editor".

P.S. The title of this post declares my intention to post a random section from my Emacs init.el files every week until I run out of things to describe. My startup Elisp has been on the web for ages, but I think that having a well-titled blog post for individual tweaks will make better Google-food, and will therefore be more useful to the world.

Two TLF quickies: social organization; prisons

One:

Tim Lee, who is rapidly proving himself one of the most unconventional (and hence interesting) libertarian bloggers, writes about markets, firms, and governments:

The central insight of [Yochai Benkler's paper] Coase's Penguin was that peer production is a form of economic organization on par with the market (first explained by Adam Smith) and the firm (first explained by Ronald Coase). Benkler expands on this tripartite classification of organizational structures in The Wealth of Networks. He spends quite a lot of time pointing out that non-market, non-firm methods of social organization account for a substantial fraction of our economic lives. We carpool, have dinner parties, give directions to strangers, help each other move, etc. These activities generate goods and services (meals, rides to work, information) that could also have been obtained via the market, but for a variety of reasons we sometimes find that non-market organizational methods meet our needs better.

I think this is a point that libertarians tend to under-appreciate. . . .

The whole post is worth reading (some decent comments too), but I want to focus on one particular part:

Progressives often think the state can convert market forms of organization into non-market, non-firm, social organization. But they're wrong. When the state gets involved, it almost always imposes a centralized, bureaucratic structure--a "firm" form of organization, in Benkler's parlance. A lot of progressive may laud the potential of public schools to create communities, but in practice, the public school system is every bit as soulless and alienating as the largest corporations. Folks on the left should hate the New York public school bureaucracy for all the same reasons they hate Wal-Mart.

This is true to a first approximation. However, I think it lumps too many kinds of "firm" together. For example, the National Science Foundation's formally a government agency, and it operates like a hierarchical "firm" in the sense that you can pull out an organizational chart and say Alice reports to Bob, who reports to Carol, who reports to Dave, etc. In another sense, however, the NSF is more highly decentralized than almost any "firm" operating in the actual market: essentially all of the NSF's "product" (i.e., scientific research) gets outsourced to universities, and a large fraction of that goes directly to individual researchers [0]. Even the job of choosing which research grants get funded is outsourced: grant proposals are reviewed for funding by committees of independent scientists.

So is the NSF truly a "firm"? Is it a "market"? A "commons"? It's hard to say. If you want to break it down, you might call it a firm (the NSF agency bureaucracy) that uses a commons/reputation economy (grant committees) to allocate resources to individual entrepeneurs (professors) who work within quasi-feudal federations (universities) of guild chapters (academic departments). Yet the NSF and its partners constitute one of the most successful forms of social organization ever devised: considered together NSF, DARPA, and the NIH may be responsible for catalyzing more basic scientific research results than any other institutions in human history. [1]

Although it may be useful to use the tools of management theory, economics, or political science to analyze the workings of this confederacy of institutions, it would be misleading to label it a market, a firm, or a commons.

More generally, although it's possible (with some intellectual contortion) to divide the economy into the "public" and "private" sectors, the institutions in each sector take radically different forms that depend on a myriad of forces (including social norms and "architecture", the other two modalities of regulation in Lessig's four-cornered graph). It could be argued that the public/private distinction often isn't the most useful discriminator when thinking about whether some form of social organization is better or worse for any particular purpose.

I would therefore characterize the progressive tendency to seek solutions in government intervention as simply a reflection of their willingness to seek solutions to problems, period. Is the market not serving a need? Maybe we should see if the government can do it. Maybe we should see if the formalized nonprofit sector can do it. Maybe we should see if peer production can do it. Maybe we should start a business. Progressives probably err on the side of too much hope for all of these, but the median libertarian seems to err on the side of dismissing all but the last as either futile, pointless, or simply immoral.

(Aside from all of the above, of course, a whole other justification for government action arises from the fact that coercive state action can sometimes solve collective action problems. But I'd rather not get into that today.)

[0] Who are, of course, then "taxed" by the university bureaucracy for overhead.

[1] Granted, these institutions have had more resources than any other scientific organizations in history, but I think we got our money's worth.

---

Two:

Adam Thierer asks why child molesters get out of jail with such short sentences. As others have noted before, the criminal justice system sets sentences based on the available capcity of prisons. According to Bureau of Justice statistics, 21% of our current prison population consists of drug offenders --- not drug-related violent crimes, which would drive this number way up, but drug trafficking, possession, etc. --- and that's trending upwards over time.

The "War" on "Drugs" wastes enormous resources that could, among other things, be used to protect children from sex criminals. Unfortunately, this kind of point's pretty tough to get across in political debate, and I can already hear in my head the shallow sound-bite rejoinder: "We don't have to choose between protecting our children from drugs and protecting them from sexual predators. We can do both:" ---pause for applause--- "we can build more prisons."

Of course, barring some revolution in incarceration policies, any more prisons that get built will be populated by roughly the same proportion of drug (and drug-related crimes) offenders, making prison-building a fantastically inefficient way to provision for improved incarceration of sex offenders. But now we're entering the territory of points too subtle to make effectively in, say, the paltry thirty seconds that were allocated for rejoinders in our last Presidential "debates". The American people might understand it, but the news media would caricature you as a wooden, uncharismatic policy wonk (if you said it calmly) or a ranting nut case (if you said it with passion). The exchange itself would last less than five minutes, but the columnists and talking heads would be repeating their talking-points caricature of it for weeks.

On confidentiality and the present forum

When I started work at Google, I signed a confidentiality agreement, as is standard at technology companies. So, of course, I now know all sorts of juicy internal Google stuff that I cannot share here, which is mildly frustrating but both obvious and expected.

What's less obvious, to me, is what I can write about. Google has an official corporate blogging policy, but it basically amounts to "don't transmit confidential information and don't tarnish Google's reputation". Now, reputation's pretty easy: reasonable people must realize that all organizations sometimes hire nutballs like me, and that therefore any random insane or obnoxious thing I say is not endorsed by Google.

However, confidentiality's a trickier matter. If I write about technology at all, then I have to do it in a way that conveys no information to the reader about confidential matters. What does that mean?

One interpretation would be that I can write about anything in technology, except when it's related to what I do at Google. But, of course, over time, that would actually reveal what I work on at Google. If I write about many topics in computing technology, but not X, then over time it becomes increasingly likely that I'm working on X, particularly if some big X-related news item comes down the pipe and I remain conspicuously silent.

(Well, lately I've been rather silent for no particular reason --- pondering my confidentiality agreement is part of it, but I also just haven't felt much like writing. But let us assume, for the sake of argument, that my posting picks up again.)

Another interpretation would be that I can't write about technology at all. In my opinion, that's not reasonable. It would be absurd for Google to ask its employees to stop thinking, speaking, and acting on technology-related issues, particularly issues related to public policy. Regardless of my source of employment, I still live in a democracy (for the time being), and therefore have both a right and a civic responsibility to participate in the public sphere. I'm assuming that Google doesn't expect its employees to forego that.

On the other hand, in a world of at-will employment, I could in principle be fired for being a good citizen. I guess I'm just betting Google won't do that.

Anyway, I could probably second-guess this forever, but at some point I have to drive a stake in the ground. So, be advised:

• If I post about X, you cannot infer that I work on anything related to X at Google, or that Google is planning to do something about X.
• If I post about X, you cannot infer that I do not work on anything related to X at Google, or that Google is not planning to do something about X.
• Anything I post here reflects my own thinking, on my own time, drawing on information that is publicly available from communication outside of any confidentiality agreement with Google, and does not reflect the existence of any Google policy, product, or snack food.

Anyway, hopefully you get the picture.

As a side note, you often hear Hayekian apologists for capitalism talk about how the market's price signals are a magnificent mechanism for aggregating information from decentralized actors. But every seriously competitive and innovative market is rife with confidentiality agreements like the one I recently signed. Price signals seem like a crude and low-bandwidth interface to such an enormous wealth of information. Astonishing amounts of knowledge gets locked up in these mile-high vertical silos, and price signals are little dime-size spigots screwed onto the bottom of each silo.

I'm not a communist. Communism was about abolishing the price signal interface. I'm a post-capitalist: I believe there must be better, richer interfaces waiting to be discovered.

I suppose you could argue that contracts already enrich the economic interface a great deal. With contracts, an economic transaction's no longer an exchange of a sum of dollars for a physical good, but rather an exchange of arbitrarily negotiable bundles of rights.

But somehow, despite that flexibility, we still end up with enormous amounts of information locked up in silos. This cannot possibly be the best way to organize information production. Academia and open source software do better in at least one way: when someone publishes a scientific result, or writes a patch for an open source project, that information gets distributed to everyone, not just the people living inside some confidentiality wall. The entire universe of potential cooperators immediately receives a richly informative broadcast of the innovation, not just a 3-tuple consisting of a scalar value and two names ("$4.95", "Alice", "Bob").

However, for a variety of complicated reasons, proprietary software companies do seem to produce certain kinds of research and software that wouldn't get made in academia or the open source world.

So, I don't really know what the answer is. I have a copy of Benkler's Wealth of Networks somewhere in my huge, freshly unpacked mound of books. I suppose I should read it.