Another name for the "wisdom of crowds"

I just read yet another online essay that, at one point, referenced "the wisdom of crowds". This phrase has been getting a lot of attention lately from the chattering classes. I don't have time to give this idea a fuller treatment, and in fact I have not read James Surowiecki's book (which started all the ruckus), but I will say something that I've wanted to get out there for a long time:

"The wisdom of crowds" is just another name for "the behavior of distributed algorithms".

When you think about it, "Let's exploit the wisdom of crowds!" really means: "Let's set up a whole bunch of independently acting, loosely federated entities, each with an incomplete view of the system, and let's make them do some cognitive task." In other words, if a crowd ends up having any wisdom, it will have arrived at it through a distributed algorithm.

Why does this matter? Two reasons.

First, it de-mystifies the concept. "The wisdom of crowds" is a phrase precisely calibrated to mystify the thing it denotes. Consider the diction: "crowds", suggesting spontaneous, informal, natural gatherings; and "wisdom", suggesting a folksy knowledge born of experience, as opposed to, say, "intelligence", "cleverness", or "expertise". The phrase "wisdom of crowds" carries within it the seeds of the message that gosh darn it, if you just got those elitist social engineers out of the way, and let everybody alone to act on their common sense, everything would be just peachy. In fact, if you read the blurbs from the publisher's page, this is exactly the message that's being pushed --- if not by Surowiecki himself, then by his promoters, with his tacit assent.

By contrast, the phrase "the behavior of distributed algorithms" is a more forbidding thing, one that highlights a crucial fact: all systems for extracting knowledge from "crowds" are, in fact, intricate constructions that achieve their results through precise engineering of the rules governing the crowd.

This leads into my second point. Any computer scientist who has tangled with distributed systems knows that designing a distributed algorithm that actually does what you want it to do is extraordinarily tricky. On the other hand, it is really easy to design distributed algorithms that, for deviously subtle reasons, end up prone to behaviors like wildly unpredictable, bizarrely pathological oscillations, race conditions, deadlock, livelock, network floods, etc., etc., etc. Until you have studied the Paxos algorithm, or at least hacked on a distributed system (and I doubt very much that James Surowiecki has done either), you probably lack the humility and skepticism needed to evaluate distributed algorithms accurately.

Naïvely lauding the alleged "wisdom of crowds" obscures the critical issue, which is the design of the distributed algorithm --- i.e., the social organization of the crowd. What are its mechanisms for passing information? For reaching consensus? Where are the possibilities for feedback loops? What happens in the obscure corner cases that result from the interactions of all its features? Etc., etc.

There's no such thing as a free lunch, and gathering together a large number of independent actors does not magically make problem-solving any easier. In fact, it can make problem-solving incalculably harder. After you gather the crowd, you have to figure out how to make it do something useful, and it is by no means the case that you'll always get acceptable outcomes by letting each individual make decisions that "look sensible" (whatever that means) based on locally available information.

Now, as I said, I have not read Surowiecki's book. It is entirely possible that I'm being utterly unfair to him based on the yammerings of others. On the other hand, the publisher's excerpt is not encouraging.

---

UPDATE 2007-09-29: If you're coming from this ycombinator blog, then note that I wrote a followup after reading most of the book and my opinion of Surowiecki himself has only marginally improved.

Also, in retrospect, it seems to me that this post is more about the "wisdom of crowds" meme --- how and why it's been successful, what's wrong with it, and the role of Surowiecki's publicist in promoting it --- than about Surowiecki's book itself.