On recent events in Hong Kong, and related lessons from American history

The BBC reports:

In China, an editorial in People's Daily warned of "unimaginable consequences" if the protests continued...

Unimaginable consequences! I literally can't imagine, actually. They can't roll out the tanks this time. In the age of ubiquitous digital photography and the Internet, it won't be a few grainy shots of one guy with a shopping bag.

I've been telling people for a while that I think that Chinese leaders are way too fearful of political protests. Look at the United States. Street protests may have affected the course of history a half century ago, but in my lifetime, street protests in America have proven to be ephemeral outbursts of emotion, more a substitute for change than a precursor of it. As I've said before, the system of government by elected representatives has evolved a nearly 100% effective immune system against alteration via protest.

For example, a hundred thousand people took to the streets of New York protesting the Iraq war. A decade later, we appear to be mired in eternal war in the Middle East. At this stage, I have serious doubts that the United States military will leave the Middle East in my lifetime. But those protestors, oh, I bet they felt heroic, like they were doing something world-changing when they were marching down that street in such massive numbers. Even that one inevitable guy holding the "Free Mumia" sign at the anti-war rally.

China's leaders need to learn to relax. Rather than clamping down on protests, they should learn from the West: create free speech zones, let the protestors expend their energy, let them lose focus and direction as the tide of life's million distractions gradually erodes their morale, and meanwhile make all the important decisions in incredibly boring committee meetings and reports, which are freely available (and sometimes even broadcast on television channels that nobody watches) but too numerous and too dull for any normal person to keep up with.

It turns out that if you construct your society just right, the drama and glamour of protests can be used, in a judo move, to undermine their effectiveness. The protest becomes a big identity-affirmation party, an end-in-itself. The shorthand "demo" (for "demonstration"), which is what many activist groups in the U.S. call a street protest, is apropos: the point becomes to show off. After the demo winds down, a bunch of energy has been expended, yet the same people are in charge and are even quite likely to be re-elected.

I don't know quite how we got here, and I'm not saying that I have a better answer, but I will note that if you told aliens that a widely held theory of change among many political activists on Earth was

1. throw a parade;
2. the government changes its behavior.

then the aliens might be confused. They might ask: what is the mechanism that you propose for this causal effect, given that the formal mechanisms for altering laws generally make no reference to parades, but rather to activities that take place mostly indoors, such as voting, drafting legislation, bureaucratic rule-making, and court decisions?

Ultimately, the Chinese government's attitude towards protest displays a deep, perhaps even touchingly naive faith in the raw power of people to change the government. It is heartwarming, in an odd way. And, all cynicism aside, I sincerely wish the Chinese people the best with the whole protest thing. Maybe they can keep it working better than we did.

On the nature of software ownership, yet again

Being a Google employee, I shouldn't comment on the specific case, but I do think this is an opportune time to repeat that by adopting software, you make a calculated bet on the future behavior of a group of people. And sometimes those people behave differently than you hoped.

There is no magic bullet that will make software developers behave the way you want indefinitely. Pay for the software with cash, pay for it with attention, pay for it with a gift economy involving the mutual exchange of labor, don't pay for it at all; make it open source, make it proprietary; eventually you will always be at the mercy of developers whose incentives converge only imperfectly with your particular desires.

Well, I should amend that: if you really really want to control your destiny, you can dedicate your life to building a software ecosystem where every user has both the freedom and the burden of control. This requires much more than open source; it requires the embrace of both Free Software and software systems designed to be just as hackable as they are usable — perhaps even more hackable than they are usable. Basically, a lot of people need to (re)read In the Beginning was the Command Line, and the story of Richard Stallman and the printer. And maybe, overall, be a little less sniffy about the more "extreme" (that is, principled and consistent) versions of the Free Software credo.

Occupy .* and the Iraq War

Then, as now, conservative opinion and elite bipartisan opinion was mostly contemptuous of the protesters. Well-fed, well-educated, well-salaried pundits looked on the shaggy protesters and remarked: how unsophisticated were the protesters' opinions, how disorganized their complaints! Fortunately, the nation was run by a select few who understood the harsh realities of a world where some suffering was necessary (for other people) so that the existing order could be maintained. And the march to war rolled on.

In all likelihood, the protests today will be as futile as those were. It's taken a couple hundred years, but the system of governance by elected representatives has evolved an immune system with nearly impervious defenses against street protests. Nevertheless in a society supported by the many and operated for the few, it is perhaps useful, for aesthetic reasons if nothing else, to have some people calling attention to that fact. If you, as a critic, imagine yourself on the side of the angels in damning the protesters, then you should perhaps reconsider.

C. Shalizi on Iraq, econophysics

Further evidence that C. Shalizi's blog, however infrequently updated, is essential reading:

• Why Oh Why Can't We Have Better Econophysics?
• The Khaldun Option

After reading a few of Cosma's essays, any person of merely ordinary erudition must wonder how he finds the time to know as much as he does. As it happens, he has a section of his FAQ devoted to this subject, and the price is steep. On the margin, however, I suspect many people would benefit from becoming slightly more like Cosma.

Expertise and journalism

Friends know that I've long held that the biggest problem with reporters is that most of them are not experts in anything in particular, which makes them gullible, easily manipulated, and prone to mangling the facts. Recently, there have been several worthwhile posts in this vein from Brad DeLong, Matt Yglesias, and (less directly) Ezra Klein.

Here's an excerpt from an actual email that I wrote two years ago, to an acquaintance who works at a major national magazine, discussing Michiko Kakutani's review of Bill Clinton's autobiography, which Brad DeLong discussed some time ago:

Moving along, if the Times prints book reviews in the daily paper, but never solicits such reviews from people who actually know something about the material in question, then that is a structural problem with the Times. Likewise if the Times rushes articles into print too soon. Perhaps I'm naive and don't know why the newspaper business is set up as it is. Perhaps there are good reasons, and not just ingrained shibboleths, causing the Times to use staff writers exclusively for the daily paper (outside of Op-Ed).

More broadly, I think the fact that journalists are mostly smart generalists is a huge weakness of the press. Save for a few geniuses, generalists are basically obsolete in every other field. Journalism should join modernity and take specialization of labor seriously, with all the attendant educational requirements. I can't count the number of times I've winced at some clueless description of computing technology in the general press. DeLong's frequent outrage at the economic ignorance (and general innumeracy) of the press shows that economics coverage is, if anything, much worse. Kakutani should not be reviewing history; she should be confined to literary fiction, where all the training one needs is to have read a great many books.

Newspapers hire full-time reporters who are not expert in any field because that has historically been an efficient way to produce news articles on deadline. At a high level, a major newspaper hires a hundred people who each write a few articles a week, instead of hiring ten thousand people who each write a few articles every hundred weeks. People assume this is simply the natural order of things, but it's not obviously the better choice: ten thousand experts can give you more substantive coverage of far more subject areas than one hundred generalists. It's true that one percent of a journalist's annual salary is a relatively small sum to pay a freelance expert for a couple of articles a year, but many experts care enough about communicating their subject matter to the public that they would probably work at cut rates (or, as blogs demonstrate, for no pay at all).

So why should newspapers be organized as small bands of generalists, instead of large federations of experts? I think the answer is coordination costs: you would need to maintain the world's biggest rolodex to keep up with all these experts, and aggregating the irregular work output of ten thousand people requires much more sophisticated planning than aggregating the regular output of one hundred people.

But coordination costs are an artifact of available technology. Mechanisms exist today that coordinate the output of many more than ten thousand voices. It's true that no existing system can serve as a drop-in replacement for the modern news reporting organization. However, the field of software-assisted content aggregation is young, and I believe that someday someone will figure out how to exploit the world's existing network of experts directly for reporting on complex issues, rather than relying on secondhand summaries of expert knowledge.

---

On a slightly related note, it's always laughable to me when I read about junk like this (and more! via Atrios; and more!), where some journalist sniffs contemptuously at the unwashed blogging rabble, and then gets promptly pwned (sorry, that truly is the mot juste).

What's especially rich to me is that, unless I misread Michael Skube's biography, Skube hasn't done hard investigative or beat reporting (at least, not full-time) for about two decades. His biography describes his freelance and city paper reporting from 1975 to 1982, but from 1982 onward his roles are described variously as an editorialist, book editor, columnist, and critic. In other words, it appears that he's spent the better part of his adult life doing exactly what he accuses bloggers of doing: sitting on his ass in a chair, reading what other people write, and commenting on it.

Two TLF quickies: social organization; prisons

One:

Tim Lee, who is rapidly proving himself one of the most unconventional (and hence interesting) libertarian bloggers, writes about markets, firms, and governments:

The central insight of [Yochai Benkler's paper] Coase's Penguin was that peer production is a form of economic organization on par with the market (first explained by Adam Smith) and the firm (first explained by Ronald Coase). Benkler expands on this tripartite classification of organizational structures in The Wealth of Networks. He spends quite a lot of time pointing out that non-market, non-firm methods of social organization account for a substantial fraction of our economic lives. We carpool, have dinner parties, give directions to strangers, help each other move, etc. These activities generate goods and services (meals, rides to work, information) that could also have been obtained via the market, but for a variety of reasons we sometimes find that non-market organizational methods meet our needs better.

I think this is a point that libertarians tend to under-appreciate. . . .

The whole post is worth reading (some decent comments too), but I want to focus on one particular part:

Progressives often think the state can convert market forms of organization into non-market, non-firm, social organization. But they're wrong. When the state gets involved, it almost always imposes a centralized, bureaucratic structure--a "firm" form of organization, in Benkler's parlance. A lot of progressive may laud the potential of public schools to create communities, but in practice, the public school system is every bit as soulless and alienating as the largest corporations. Folks on the left should hate the New York public school bureaucracy for all the same reasons they hate Wal-Mart.

This is true to a first approximation. However, I think it lumps too many kinds of "firm" together. For example, the National Science Foundation's formally a government agency, and it operates like a hierarchical "firm" in the sense that you can pull out an organizational chart and say Alice reports to Bob, who reports to Carol, who reports to Dave, etc. In another sense, however, the NSF is more highly decentralized than almost any "firm" operating in the actual market: essentially all of the NSF's "product" (i.e., scientific research) gets outsourced to universities, and a large fraction of that goes directly to individual researchers [0]. Even the job of choosing which research grants get funded is outsourced: grant proposals are reviewed for funding by committees of independent scientists.

So is the NSF truly a "firm"? Is it a "market"? A "commons"? It's hard to say. If you want to break it down, you might call it a firm (the NSF agency bureaucracy) that uses a commons/reputation economy (grant committees) to allocate resources to individual entrepeneurs (professors) who work within quasi-feudal federations (universities) of guild chapters (academic departments). Yet the NSF and its partners constitute one of the most successful forms of social organization ever devised: considered together NSF, DARPA, and the NIH may be responsible for catalyzing more basic scientific research results than any other institutions in human history. [1]

Although it may be useful to use the tools of management theory, economics, or political science to analyze the workings of this confederacy of institutions, it would be misleading to label it a market, a firm, or a commons.

More generally, although it's possible (with some intellectual contortion) to divide the economy into the "public" and "private" sectors, the institutions in each sector take radically different forms that depend on a myriad of forces (including social norms and "architecture", the other two modalities of regulation in Lessig's four-cornered graph). It could be argued that the public/private distinction often isn't the most useful discriminator when thinking about whether some form of social organization is better or worse for any particular purpose.

I would therefore characterize the progressive tendency to seek solutions in government intervention as simply a reflection of their willingness to seek solutions to problems, period. Is the market not serving a need? Maybe we should see if the government can do it. Maybe we should see if the formalized nonprofit sector can do it. Maybe we should see if peer production can do it. Maybe we should start a business. Progressives probably err on the side of too much hope for all of these, but the median libertarian seems to err on the side of dismissing all but the last as either futile, pointless, or simply immoral.

(Aside from all of the above, of course, a whole other justification for government action arises from the fact that coercive state action can sometimes solve collective action problems. But I'd rather not get into that today.)

[0] Who are, of course, then "taxed" by the university bureaucracy for overhead.

[1] Granted, these institutions have had more resources than any other scientific organizations in history, but I think we got our money's worth.

---

Two:

Adam Thierer asks why child molesters get out of jail with such short sentences. As others have noted before, the criminal justice system sets sentences based on the available capcity of prisons. According to Bureau of Justice statistics, 21% of our current prison population consists of drug offenders --- not drug-related violent crimes, which would drive this number way up, but drug trafficking, possession, etc. --- and that's trending upwards over time.

The "War" on "Drugs" wastes enormous resources that could, among other things, be used to protect children from sex criminals. Unfortunately, this kind of point's pretty tough to get across in political debate, and I can already hear in my head the shallow sound-bite rejoinder: "We don't have to choose between protecting our children from drugs and protecting them from sexual predators. We can do both:" ---pause for applause--- "we can build more prisons."

Of course, barring some revolution in incarceration policies, any more prisons that get built will be populated by roughly the same proportion of drug (and drug-related crimes) offenders, making prison-building a fantastically inefficient way to provision for improved incarceration of sex offenders. But now we're entering the territory of points too subtle to make effectively in, say, the paltry thirty seconds that were allocated for rejoinders in our last Presidential "debates". The American people might understand it, but the news media would caricature you as a wooden, uncharismatic policy wonk (if you said it calmly) or a ranting nut case (if you said it with passion). The exchange itself would last less than five minutes, but the columnists and talking heads would be repeating their talking-points caricature of it for weeks.

Another name for the "wisdom of crowds"

I just read yet another online essay that, at one point, referenced "the wisdom of crowds". This phrase has been getting a lot of attention lately from the chattering classes. I don't have time to give this idea a fuller treatment, and in fact I have not read James Surowiecki's book (which started all the ruckus), but I will say something that I've wanted to get out there for a long time:

"The wisdom of crowds" is just another name for "the behavior of distributed algorithms".

When you think about it, "Let's exploit the wisdom of crowds!" really means: "Let's set up a whole bunch of independently acting, loosely federated entities, each with an incomplete view of the system, and let's make them do some cognitive task." In other words, if a crowd ends up having any wisdom, it will have arrived at it through a distributed algorithm.

Why does this matter? Two reasons.

First, it de-mystifies the concept. "The wisdom of crowds" is a phrase precisely calibrated to mystify the thing it denotes. Consider the diction: "crowds", suggesting spontaneous, informal, natural gatherings; and "wisdom", suggesting a folksy knowledge born of experience, as opposed to, say, "intelligence", "cleverness", or "expertise". The phrase "wisdom of crowds" carries within it the seeds of the message that gosh darn it, if you just got those elitist social engineers out of the way, and let everybody alone to act on their common sense, everything would be just peachy. In fact, if you read the blurbs from the publisher's page, this is exactly the message that's being pushed --- if not by Surowiecki himself, then by his promoters, with his tacit assent.

By contrast, the phrase "the behavior of distributed algorithms" is a more forbidding thing, one that highlights a crucial fact: all systems for extracting knowledge from "crowds" are, in fact, intricate constructions that achieve their results through precise engineering of the rules governing the crowd.

This leads into my second point. Any computer scientist who has tangled with distributed systems knows that designing a distributed algorithm that actually does what you want it to do is extraordinarily tricky. On the other hand, it is really easy to design distributed algorithms that, for deviously subtle reasons, end up prone to behaviors like wildly unpredictable, bizarrely pathological oscillations, race conditions, deadlock, livelock, network floods, etc., etc., etc. Until you have studied the Paxos algorithm, or at least hacked on a distributed system (and I doubt very much that James Surowiecki has done either), you probably lack the humility and skepticism needed to evaluate distributed algorithms accurately.

Naïvely lauding the alleged "wisdom of crowds" obscures the critical issue, which is the design of the distributed algorithm --- i.e., the social organization of the crowd. What are its mechanisms for passing information? For reaching consensus? Where are the possibilities for feedback loops? What happens in the obscure corner cases that result from the interactions of all its features? Etc., etc.

There's no such thing as a free lunch, and gathering together a large number of independent actors does not magically make problem-solving any easier. In fact, it can make problem-solving incalculably harder. After you gather the crowd, you have to figure out how to make it do something useful, and it is by no means the case that you'll always get acceptable outcomes by letting each individual make decisions that "look sensible" (whatever that means) based on locally available information.

Now, as I said, I have not read Surowiecki's book. It is entirely possible that I'm being utterly unfair to him based on the yammerings of others. On the other hand, the publisher's excerpt is not encouraging.

---

UPDATE 2007-09-29: If you're coming from this ycombinator blog, then note that I wrote a followup after reading most of the book and my opinion of Surowiecki himself has only marginally improved.

Also, in retrospect, it seems to me that this post is more about the "wisdom of crowds" meme --- how and why it's been successful, what's wrong with it, and the role of Surowiecki's publicist in promoting it --- than about Surowiecki's book itself.