Showing posts with label business. Show all posts
Showing posts with label business. Show all posts

Monday, December 04, 2017

Two ways ISPs can do content-based filtering of encrypted traffic

Vaguely a propos of the revived net neutrality debate, a while back I saw someone on Twitter claim that it is technically not possible for ISPs to do content-based (as opposed to destination-based) filtering of SSL traffic. This statement seems initially plausible, but is false. I can think of two technical mechanisms to do content-based filtering.

First, it is possible to identify encrypted content via traffic analysis. ISPs could compile a database of traffic signatures which they wish to throttle (e.g., for videos that are available from their own video streaming services) and throttle any traffic matching that signature.

Second, ISPs can require that users add a trusted SSL root cert owned by the ISP, thus allowing the ISP to man-in-the-middle all SSL traffic. Obviously, content-based filtering then becomes trivial.

You might object that this second measure would be unacceptably onerous, and would be rejected by the market. In the near future, a middle-class American family of four may own ten or twenty Internet-connected devices, running a half-dozen operating systems, and demanding that users install a root cert on all of them would cause unbelievable inconvenience and outcry. This might be true, but without even trying very hard I can think of numerous ways that ISPs could try to acclimate users to this bitter pill:

  • Of course, the software package would be named something relatively innocuous, like "Comcast Internet Security Accelerator" or some such nonsense.
  • The MITM cert might only be required for devices that wish to access the "fast lane" — in other words, the ISP would simply throttle any SSL connection that it does not MITM. All the household's devices would be functional even if you didn't jump through this hoop, but the ones that need the fastest connections — say, the PC that streams HD VR video — would require the MITM cert installation.
  • The ISP could distribute web browsers and other apps that embed the trusted cert — for example, Comcast could provide a custom build of Chromium — and require their use for the "fast lane". Again, you wouldn't need this app for casual web browsing, only for sites that are sensitive to speed.
  • ISPs could strike distribution deals with mobile carriers to install root certs on phones. The most obnoxious way to do this would be to ship the phone's ROM with the MITM cert baked in; this would probably cause massive outcry, akin to the eDellRoot debacle. A sneakier way to do it would be to ship a carrier-branded app that has the ability to update the trusted cert store (by itself this is arguably innocuous), along with an ISP-branded app that (a) nags the user for consent when it detects that the phone is on the ISP's network, something like "Welcome to Comcast! Do you want to enable Comcast Fast Lane[TM]?", and (b) when the user "consents", installs the MITM root cert by delegating to the carrier's app.
  • ISPs could embed a web browser connected to a virtualized display in the set-top box. The set-top box, of course, would already trust the MITM cert. Then, instead of browsing directly to https://www.youtube.com/ or whatever, you would first browse to http://xfinity.local/, which would present you with a web app that is itself a browser running via remote desktop protocol. Then you would type https://www.youtube.com/ into the address bar of this web browser. The ISP could even "helpfully" set up its DNS to perform this redirection automatically (if you type youtube.com without the https).

These are just the ideas that occur to me in about twenty minutes of thinking. If these seem farfetched to you, there may be other ways to boil this frog. Companies can be rather creative when there are billions of dollars of rents to be extracted. The result does not have to be low-cost or seamless for the user; local broadband ISPs in the United States are subject to practically no competition and whatever they implement just has to be marginally less painful than waiting for your content to download over the cellular network.

Friday, September 29, 2017

Tentpole sponsors: an idea for improving paid service virality

Ad-supported communication platforms like Facebook have many structural advantages over hypothetical competitors that charge users money directly. One advantage is that a purely ad-supported service can spread virally, from user to user, at a vastly greater rate than a service that demands direct monetary payment.

For most users, the unpredictable, frequently unmeasurable harms of losing privacy and control over their social identity are less tangible than the direct time and money cost of signing up for a paid software subscription [0]. Thus free services which strip-mine your privacy and lock you into their prison spread like wildfire, while paid services that respect their users barely get off the ground. It seems that every large social networking service on the Internet has been hammered on the anvil of this seemingly inescapable logic and beaten into a Facebook-like shape.

However, user preferences vary. One can conjecture that within any social network subgraph of size N (for some N), there exists at least one user who cares an unusual amount about privacy and control. This user might be willing to subsidize a large subset of their local subgraph. Let R be the ratio of the local neighborhood of size N that such a user is willing to subsidize.

If N and R have the right values, a possible hack for the virality problem is to charge money to these special users — call them "tentpole users" — and allow them to sponsor the addition and ongoing use of the users around them. Most users will not be tentpoles; but given enough poles, positioned appropriately, the tent may be lifted over the entire addressable user population.

In the most basic form, you can imagine that a paid subscription gives every user a certain number of tokens, which they can use to sponsor accounts for their friends and family. When a new user is invited, some tokens would be allocated to them — one to support that user, and optionally some extra tokens gifted so that they could invite more users in turn. A non-sponsor user who wants additional invitations beyond their starter set would purchase more, thereby becoming a sponsor, or ask their network for some spare tokens. Sponsorship would be fungible — that is, users would be able to change their sponsor at any time — but every user would be either a sponsor or a beneficiary or both.

In principle, with proper tuning, most users could be beneficiaries, and pay nothing. A service engineered this way would be closer in virality to an ad-supported one. (It's still not quite as viral; for one thing, there is still some real friction at the edge of the "sponsorship radius", the distance from a sponsor at which users run out of tokens for further invitations. This needs further thought.)

Another model would allow all users to join free of charge, but grant additional privileges to sponsored users. This works, economically, as long as the aggregate cost of free-riding users is less than the total revenue from sponsors. This "tentpole freemium" model resembles an ordinary freemium model (where only the sponsors themselves pay [1]); arguably it is simply a freemium model where one of the premium benefits is improved amenities for one's contacts.

When I mentioned these ideas to a colleague a few months ago, he immediately pointed out that tentpoling leads to a situation where sponsored users are socially indebted to their sponsors. This has at least two effects. First, debt potentially causes social awkwardness, and this risk must be navigated (c.f. V. A. Zelizer). Second, users may feel a sense of precarity because sponsorship could end (for example, if their sponsor cancels their subscription), and thus would be reluctant to adopt the platform. These are definitely challenges, but it may be possible to overcome them.

Social awkwardness may be amenable to psychological hacks which obfuscate the transactionality of the interaction. To invent a silly example, one can imagine a social network where your profile picture can be decorated with a virtual hat, which degrades over time. You can only remain on the service if your profile has a hat; sponsors receive a certain number of hat credits, which they can use to purchase various hats and gift them to their peers. Lastly, any user can trade or gift a hat that they possess. The combination of these mechanics makes the act of "wearing" a hat expressive, not merely pecuniary; wearing a hat that one of your friends obtained and gave to you can be construed as a fun social act which strengthens your friendship, rather than a purely financial necessity. By adjusting the number of hat credits that sponsors get, you can create enough liquidity in the system that most active users have multiple hats. Therefore, it is possible to beg your friends for a particular hat without disclosing that you just don't feel like buying any hats — for example, a user who doesn't want to pay for the service might ask "Hey, anybody got a spare blue knit cap? My last one is expiring next week." A certain degree of strategic ambiguity is preserved.

This example is crude and probably too nakedly gamified to work, but I hope it illustrates that there is a gigantic space of possibilities for designing the social character of sponsorship. Somewhere in that space, I conjecture that there is a point where people are comfortable with sponsor-beneficiary relations in a social network.

Precarity may also be amenable to engineering solutions. For example, one could allow and encourage users to be sponsored by multiple people, and then grant enough tokens to sponsors that their "radius of influence" would, in practice, always overlap with other sponsors'. Then, in steady state, most users would feel secure, because they would be sponsored by more than one person. And in a tentpole freemium model, users would always continue to have access to their identity even when sponsorships expire, reducing the downside even if one were to lose all of one's sponsors.

Have there been examples of tentpole sponsorship as a business model in the wild? I have trouble thinking of them.

Anecdotally, one sometimes hears of people buying paid Slack workspaces to socialize or organize activities that are not part of their day job. I assume that there are usually free riders in this arrangement. So, Slack may have stumbled on this model without intending to (obviously, Slack's primary revenue stream is charging businesses for employee accounts, which is socially a very different scenario, although arguably isomorphic to tentpole sponsorship in some ways).

Alternatively, one could argue that whenever a highly technical user sets up a custom email domain for their family, rather than just signing everyone up for Gmail, they are tentpoling the base protocols of the Internet. The difference, I guess, is that sponsorship is not fungible: if you set up a domain for your family, your child cannot change their sponsor later in life without migrating to another domain, which incurs various transition costs.

The last example I can think of is in gaming. In some multiplayer games like Lineage, players can organize into clans, and clans can purchase in-game collective goods. I've never played Lineage, but I assume that players within a clan differ in their level of contribution, and thus the most committed players are effectively sponsoring the rest.

Overall, however, I think the idea of tentpole sponsorship has seen little use, and this seems like a space that is ripe for experimentation.


Having read this, your reaction might be (probably should be!), "Talk is cheap. Ideas are cheap. What are you gonna do about it?"

Alas, I have to admit that the answer is very little.

To really pursue this idea would be multi-year effort, and there are all kinds of reasons that this does not seem like the thing that I want to spend the next few years building. (For one thing, a half-hermit misanthrope like me is probably one of the worst people in the world to try building a social network.) So, instead, I'm throwing this post out there in a sort of cry to the universe, both to get it out of my head, and also in the vague hope that it infinitesimally increases the probability that somebody will figure out how to make it work.

This may be the dumbest theory of change that's ever been written down, but it's about what I can muster at this point in my life. On the other hand, if you back up and squint, in 2009 I predicted (sort of) both the business model of Patreon and Jeff Bezos's purchase of the Washington Post, so maybe the universe will again cough up something resembling my half-baked ideas.


Bonus thought: once you have the idea of tentpoling in your mental toolkit, you will begin to see echoes of it in many places. For example, nearly every software package is sometimes hard to use. But some users have the inclination and capability to become expert in that software, and then spend effort helping others cope with it. These helpful experts are technical (rather than financial) tentpoles, paying the cost of onboarding and support for users in some radius around them. Every geek who serves as tech support for their parents' devices is holding up the tent of Microsoft or Apple or Google or whatever over their family.

In fact, many instances of free riding can be thought of as tentpoling on some level. I suppose the difference between the concept of tentpoling and free-riding in general is that tentpoling is voluntary and has a significant dimension of locality in the social graph.


[0] Arguably, there is also a market in lemons for software services that offer users privacy and control. This is a separate issue and much too big to tackle in this post.

[1] On a vaguely related note, observe that Maciej Ceglowski has repeatedly suggested that Twitter should adopt an ordinary-freemium model where users just pay money for additional features. It is an interesting thought puzzle to contemplate why Twitter has never even experimented with doing this. There seems to be a real organizational dynamic in business that once a company settles on an advertising-supported revenue model, this sucks up all the oxygen necessary for alternate revenue models to breathe, and I do not entirely understand why. Consider how long it took for YouTube to offer YouTube Red; although this is also a case which proves that it is not impossible for the alternative model to break through.

Monday, September 25, 2017

What's the point of Facebook alternatives?

It is clear at this point that Facebook has a monopoly on online human-to-human interaction that no private forces, market or otherwise, will break in the foreseeable future. The network effects from a billion users are unsurmountably large. If we take Metcalfe's Law literally, even a social network that accumulates a hundred million users will be a hundred times less powerful than Facebook.[0] In fact, you're probably confused by the title of this post: What Facebook alternatives?

Facebook is furthermore unlike the other American technology giants in that it alone locks up all its users' interactions inside its walled garden. Apple, Alphabet, Amazon, and Microsoft are, to greater or lesser degree, porous at the edges — you can use an iPhone to chat with people who don't have iPhones; you can use Gmail to email people who don't have Gmail; buying things from Amazon doesn't prevent you from buying other stuff elsewhere; even Microsoft has realized belatedly that it is not the center of the universe & its products have started playing nice with others. But Facebook locks up your posts, locks up your photos, locks up your entire social identity inside its prison. There simply is no way to interact with Facebook users except by creating a Facebook account yourself and creating content that further entrenches Facebook's monopoly.

The gradual decay of open Internet protocols as human interaction disappears down the black hole of Facebook's ever-expanding digestive tract has been one of the great disappointments of my lifetime. In the end, AOL seems to have beaten the Internet after all.

I have opted for only de minimis engagement with Facebook, and more or less refuse to communicate via its platform. This has probably attenuated some of my relationships with people in a regrettable way (if you're somehow reading this and you wish this hadn't happened between us, send me email! it still works!) but the actions of conscientious objectors like me have not made the tiniest scratch on Facebook's dominance.

It is only a matter of time before governments realize that this entity must be regulated, whether under antitrust law or otherwise. The question is what will happen then.

In my opinion, it is clear what the ideal outcome would be: forcing Facebook to adopt open APIs that give users transparency, portability, and interoperability. Users should be able to see the data that Facebook has stored about them. Users should be able to export that data in toto to competing platforms. And users should be able to interoperate between Facebook and other social networks — a future version of Diaspora*, for example, should be able to see and interact with Facebook content generated by that Diaspora* user's social network, and vice versa; interactions between users across platforms should be reflected accurately on both sides. A user would thus be able to leave Facebook without severing their ties to the users they have left behind.

In a world where these APIs existed, users would have a way to reject Facebook's toxic business model and questionable privacy practices without exiling themselves from their social life. In Hirschmanian terms, users would have the option of exit, not just voice, as a way of signaling dissatisfaction. Facebook would probably even get healthier, as a product, as a result of the opportunity for meaningful competition.

This outcome is exceptionally unlikely. Government regulation of Facebook, although likely inevitable, is also likely to be ham-fisted and ineffective, simply because governments are terrible at understanding technology and rarely have the political will to impose effective solutions even if they knew of them. The last time the U.S. government, for example, used antitrust law against a technology monopolist, it basically bogged down the company in red tape for a decade but did little to meaningfully give its competitors an opening in the allegedly monopolized market. Windows is still by far the most widely used desktop operating system and the web browser that finally dethroned Internet Explorer on Windows did so through incredibly aggressive Internet marketing, not by using the remedies forced on Microsoft by antitrust law.

However, there is one thing that technologists might be able to do to make the desirable outcome marginally more likely, and that is to develop the protocols, and plausible implementations thereof, that would allow effective federated social networking to be mandated by government decree. Diaspora* may have made a significant dent in a subset of the technical problems, but there are significant open challenges inherent to federated social networking that I suspect have not been solved.

Critics of Diaspora*, Mastodon, etc. thus misstep when they observe that organic growth of these platforms is limited. The ultimate destiny of a successful federated social networking protocol, if one ever arises, will be to stock the toolkit of a future regulator, not to overtake Facebook via organic growth.

There is enormous inertia—a tyranny of the status quo—in private and especially governmental arrangements. Only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around. That, I believe, is our basic function: to develop alternatives to existing policies, to keep them alive and available until the politically impossible becomes politically inevitable.

— Milton Friedman, Capitalism and Freedom


[0] 100M is, to an order-of-magnitude approximation, the size of Snapchat's active user population. Observers who think Snapchat is a credible challenger to Facebook are off by a factor of a hundred, not a factor of ten.

Thursday, September 18, 2014

US ISPs do not deserve much credit for increasing broadband speeds

Disclaimer: I worked for Google from 2006-2013, although not on Fiber.

Towards the end of this Twitter thread sparked by Timothy B. Lee, a commenter writes (by way of defending US ISPs):

Internet speeds have increased 1500% in ten years. 250k Wi-Fi hotspots are now available. That's progress.

When I read this, I just thought, this is such bullshit. Taking it apart with sufficiently satisfying thoroughness requires more than 140 characters, so I'm going to say it here.

US ISPs are riding the wave of technological progress. Giving them credit for the wave is evidence of cluelessness so extreme as to strongly suggest intellectual dishonesty.

First, crediting ISPs for the spread of wireless hotspots is especially egregious: it's a bit like giving them credit for Moore's Law. Even if ISPs had completely failed to increase speeds beyond dial-up, people would still want local networks without the hassle of Ethernet cables. The development and spread of wireless technology was not driven by ISPs. In fact, in some ways the opposite is true, as many ISPs retained official policies against running an open wireless hotspot (or even connecting multiple devices via NAT!) long after broadband became widely available.

Second, as for a 15x improvement in ten years[0]: that might sound impressive to some people, but all I can think is that ten years equals nearly seven doublings of Moore's Law and 27 = 128. Network speed doesn't track transistor density exactly, but computing technology is full of exponential curves like this (storage density, for example, doubles even faster than Moore's Law). To anyone with a clue about computing technology, 15x in 10 years obviously sounds somewhere between mediocre and lousy. In fact, Nielsen's Law predicts compound improvement of 57x over 10 years, or nearly 4x the observed improvement claimed by Dietz.[1] When Dietz calls out 15x improvement as a talking point in ISPs' favor, he is trying to rhetorically exploit an information asymmetry, namely his audience's presumed ignorance of exponential curves in computing technology.

Therefore, the reality is that US ISPs are badly managed technological laggards, just like everyone thinks.

In fact, the pace at which ISPs have taken up technological innovation is so bad that an entrant from another industry was able to enter the market and, without even trying very hard, spank the incumbents so thoroughly that it became a national embarrassment.

Google does so many things so well that you may not even be properly surprised by this fact. Let me try to give you a visceral feel for how anomalous Google Fiber is. Consider what happened when Google entered a market where its major competitor really was pushing innovation to the limit: consider Android. Google had to dedicate hundreds of its best and hardest-working engineers to the project and enlist the support of essentially every other company in the industry, and after eight hard-fought years, the best it can show for its efforts is rough technological parity with Apple.

Or, to take an example where Google was the defender, consider what happened when Microsoft decided to enter the search engine market. Again, this is a market where the strongest competitor really was pushing innovation to the limit. Billions of dollars and millions of engineer-hours later, Bing's search quality is still slightly worse than Google's.

I don't know the head count for Google Fiber (and even if I did, it would be covered by my NDA) but I will venture a strong guess that its engineering head count is far less than Android's, like at least an order of magnitude. In Google's product portfolio, judging by the scale of investment and strategic value, Google Fiber is basically a hobby, one that Google would never even have tried if US ISPs were remotely as good at their jobs as, say, ISPs in South Korea. And yet, technologically, Google Fiber simply outclassed these incumbents, who are supposedly competing and innovating furiously to earn your dollar.[2]

Imagine you had a friend who claimed to be training super-hard at tennis. But whenever you're at the courts, you see them go out to the court, set up the tennis ball machine, and swat lethargically at balls for about ten minutes, after which they sit down to sip an energy drink and diddle around on their phone. Then, one day, your niece, who's in reasonably good shape but has never picked up a tennis racket in her life, drops by, and she walks onto the court and crushes your friend in straight sets without breaking a sweat. You might reasonably question whether your friend was really training as hard as they claim.

In short, lauding the big US ISPs for their piddling achievements is the soft bigotry of low expectations.

But don't worry, ISPs, America isn't giving up on you. We believe that if you're subjected to the right mixture of tough love, including the right kinds of competition and regulation, you're capable of achieving just as much as Google Fiber did. Or, at a minimum, we believe that the ISP market is capable of achieving that. In the meantime, we'll try hard to ignore the risible bullshit of telecom hacks who claim that you're doing well enough.


[0] Engineers generally use multiples rather than percentages to describe improvements of this magnitude. Dietz's use of "increased 1500%" rather than 15x is a classic PR hack's way of making modestly sized numbers seem gigantic.

[1] Nielsen's predictions are fitted to observations, but those observations are for "high-end users". Dietz's number, which presumably describes mean or median users, reveals how poorly ISPs have done at making high-speed internet affordable despite the march of technology. Note that we do not see this failure in other, more competitive areas of computing technology: Intel's mid-range notebook-class processors follow Moore's Law just as well as its high-end server chips.

[2] This paragraph contains a logical honeypot for telecom hacks, who are likely to see an argument that they think they can debunk, but which they can only debunk by resorting to utter bullshit, which they will promptly be called upon. See if you can spot it.

Friday, August 29, 2014

Uber vs taxi wait times: non-anecdotal evidence

More on the taxis vs. Uber in San Francisco beat.

I repeat my previous assertion that, anecdotally, ">20 min" is an understatement on weekend nights, and that the decision to use only 3 buckets ("<10 min", "10-20 min", ">20 min") conceals the long tail. As in so many other applications, tail latency has a disproportionate negative effect on on user experience, and looking at even the median or mean latency is insufficient: you should measure out to 95th percentile latency at least (n.b. this is standard procedure at Google).

Monday, June 23, 2014

More on taxis

I will fully cop to the fact that my previous bitching about taxis concerned a classic San Francisco yuppie firstworldproblem but here are some more stories about how taxis in many cities serve people of all stripes exceptionally poorly.

Relatedly, I took an UberX with a friend the other night and the experience was awesome.

I might have liked the option of tipping the the driver (via the app, after you get out of the car), but UberX doesn't allow that. On the other hand, maybe tipping would establish a social norm for companies like Uber to underpay their drivers with the expectation of tips? I really like traveling abroad in countries that don't have a tipping culture; it seems more rational and arguably in the long run leaves labor in a better position since their compensation is assured by contractual terms rather than manners. So maybe it's great that Uber doesn't have tipping, only reviews.

Wednesday, June 18, 2014

What happens when you let people pay to remove ads

Disclaimer: Obviously I worked at Google for 6.5 years so take this with the appropriate grain of salt.[0]

"I wish Google would just let me pay them money to remove ads from this service. WTF GOOGLE IS SO DUMB AND EVIL WHY WON'T THEY DO THIS."

How many times, in various corners of the Internet, have you read some variation of the preceding sentiment?

Now, it turns out, Google is trying to do this with a subset of music videos on YouTube. But when you do this, you need a contract with the rights holder. This is complicated. The rights holder might not agree to your contract terms. For example, they might want more money. So you end up in negotiations. While those negotiations are ongoing, or if they break down, you can't include that music in your paid service.

The Internet's collective reaction has been: "WTF GOOGLE IS SO DUMB AND EVIL WHY ARE THEY DOING THIS?"

Now, YouTube will still host music videos that are not opted into its monetization program, i.e., if you are a band and want to put music videos on the Internet for free, YouTube will host those videos for you. Let's be clear what this means: if you want to distribute a video to a hundred million strangers on the Internet, YouTube will pay for the software and servers and a petabyte of network bandwidth and a small army of SREs holding pagers who will wake up at 3am if too many people in Kuala Lumpur click on your video and get an HTTP 50x error, and it will do so without you paying YouTube a dime. None of that's changing. But if you opt into YouTube's monetization program, you will have to opt into its full, updated monetization program: ads for non-subscribers, and no ads for subscribers.

Note that there is no sensible way to let users pay to remove ads from music videos while also still showing ads to those same users for some music videos. If YouTube did this, you can bet the Internet would collectively scream, once again, "WTF GOOGLE IS SO DUMB AND EVIL WHY DON'T THEY REMOVE ADS ON MUSIC VIDEOS WHEN I PAID TO HAVE ADS REMOVED? I'm unsubscribing from this bullshit service!" The subscription service would fail and YouTube would have to revert to its old model of monetizing via ads only.

In short, "blocking" — i.e., excluding unlicensed music from the monetization program — is an inevitable consequence of having a paid subscription service.

The press has done an abysmal job covering this. It seems that every year my contempt for (most) journalists finds reasons to grow greater and greater. I'm pretty disappointed that the Financial Times, reputedly pretty reliable, appears to be Ground Zero for this particular blast of misinformation.

p.s. As for claims that YouTube has anything approaching a monopoly on online video sharing, I'm honestly puzzled by the claim. Just to take one example competitor, Vimeo is pretty reliable and seems to have rough feature parity, including embedding in external sites. Vimeo's content is included in Google's video search corpus (example) and therefore shows up as a thumbnail image in Google web search, just like YouTube videos. If you need to make money, Vimeo has a few built-in monetization options; if those are insufficient, there's a lot of innovation occurring in the world of funding, and in my opinion the Patreon/Subbable model seems much more promising for creators with a small-to-medium-sized audience than Spotify-style monetization, which currently amounts to "we'll give you 0.0001 pennies per stream play so that both the company and the artists can lose money hand over fist!" Is it really true that Adele fans will face substantial (or even non-substantial) barriers to watching her music videos if they're on Vimeo? And that's just one competitor. The upshot is that indie labels are probably wrangling with YouTube over licensing terms not because it has anything like a monopoly, but because they think wrangling with YouTube will make them more money than all the other alternatives. Which is perfectly fine — more power to them, negotiating the best deal is essential in business, etc. — but we should not misread the situation.


[0] On the other hand, one benefit of no longer working there is that I can write stuff like this post, just like I used to before I worked there. Arguably, working at Google made me less predisposed to harshly criticizing misinformed critics of Google.

Uber and taxis in San Francisco

Uber is a national story now, but this post from a former colleague reminded me of a little fact about Uber's origins that many observers outside of San Francisco probably don't know.

Before Uber existed, San Francisco taxicabs were extremely unreliable. On busy nights, you would commonly call a cab dispatching company on your cell phone, talk to a dispatcher who promised that a cab would arrive, wait half an hour or more, and still have no cab show up. This only has to happen a couple of times, on a chilly San Francisco Saturday night while your date stands on the street shivering in her dress and heels, before you start saying to yourself, "Fuck all taxicab companies, forever."

Conversely, I speculate that it was not uncommon for cab drivers to receive a call, only to have the passengers disappear, since they'd grabbed another cab that happened by sooner. In fact, once I concluded that taxi drivers were as likely to flake out on me as not, I started doing this myself. What else could you do?

In other words, game-theoretically, taxicabs and passengers were stuck in a low-trust equilibrium. Passengers could not rely on cabs, so they would grab any cab they could, even if they'd called one. Cabs could not rely on passengers, so they would pick up any fare that looked promising, even if they had been dispatched. A taxi company could have disrupted this equilibrium by offering reliable service and building a brand name as such. None of them did. Much like your local cable company, they were collectively content to sit on their government-protected oligopoly and treat customers like garbage.

(Notice that passengers can't disrupt this equilibrium: there's no equivalent "union of passengers" to which a reputation for reliability could be attached.)

The most important thing about Uber, at least initially, was not that you called it with a handy smartphone app, or that Uber cars were fancy black limos, nor even that they always accepted credit cards.[0] It was simpler than that: Uber cars came when you called. Uber achieves this result through a variety of means, but the most obvious is simply holding individual drivers accountable for every ride.

Therefore, at least in San Francisco, the whining of taxi companies exposed to competition from Uber and Lyft is the whining of any industry that serves its customers badly, and then is exposed to superior competition that serves its customers well.

Uber's initial success in San Francisco was a key stepping stone in their rise to national prominence. One wonders whether Uber would exist today if even a single SF cab company had served customers reliably.


[0] p.s. The credit card thing. Every SF taxi rider has had this happen: the driver has the credit card reader in the car, staring you in the face, but the driver says: "Sorry, it's not working." Now, if this happened once in a while, you might believe it. But it happens every single time. Is it really the case that every single credit card reader in the SF taxicab fleet is out of order 100% of the time despite a good-faith effort to keep them operational? Or is it much more likely that the driver is trying to cheat on his taxes, which is made easier if he forces all his customers to tip him in cash? Or else the taxi company is cheating on its obligations to its drivers by keeping tips paid on credit cards for itself? And now this confederacy of cheats wants the government to protect them from competition? Do you feel sympathy for this band of cynical hypocrites?

Sunday, April 15, 2012

Stross on ebooks

Disclaimer: Since 2006 I have been employed by Google, which sells ebooks and related technology, and therefore competes with several of the companies involved in the subject matter. My opinions are my own and not those of my employer.

Charlie Stross has a compelling analysis of the book publishers' position, in light of the recently opened Department of Justice lawsuit against Apple for alleged price fixing. (Incidentally Stross's comments are usually decent and he engages with his readers as well, so they're worth a skim.)

Stross's post reminds me a lot of something I wrote about 2 years ago. Rereading that post today, I find that I have very little to add to it.

There is one additional thing from Stross's post, though, that I find particularly galling:

. . . if your boss is a 70 year old billionaire who also owns a movie studio and listens to the MPAA, you don't get a vote. Speaking out against DRM was, as more than one editor told me over the past decade, potentially a career-limiting move.

Publishing companies like to portray themselves as scrappy underdogs locked in heroic battle on the side of knowledge against the forces of ignorance. In fact, they are merely subordinate tentacles of large, stupid media conglomerates; they aid the forces of knowledge when it is convenient for business, and do whatever they can to muzzle open discourse when that is convenient. (Fortunately their power to silence critics is fairly limited in a free society.) Critics have been repeating for years that ebooks should be convenient and DRM-free. Publishers never listened; instead they threatened the careers of people like Stross's editors for even bringing it up.

Anyway, I can't comment on the substance of the DoJ lawsuit (and anyway, I know nothing about the facts of the case, so my comments would be pointless), but I find it hard to muster much sympathy for these people. The solution to their problems has been staring them in the face for as long as ebooks have existed. Unfortunately, I'm somewhat less sanguine than Stross that publishers are going to learn the lesson. They seem pretty impervious to persuasion.


p.s. See also Tim O'Reilly's Plus post.

Friday, April 13, 2012

The ideology of Dilbert

So apparently people are still leaving comments on that post about Dilbert from half a decade ago. The latest calls me a pointy-haired boss (this is a little funny because I've never been a manager; on the other hand you could make the case that PHB-ness is a state of mind rather than a job title). I've been arguing with people on the Internet for so long, and am consequently so thick-skinned, that insults as mild as that one barely even register; but anyway the ping prompted me to think even more about how Dilbert sucks.

Aside from failures in basic craftsmanship, which I discussed in my previous post, Dilbert cultivates a poisonous worldview. Here are a few lessons that you will learn by reading Dilbert strips:

  • Your boss, co-workers, and clients are not human beings, but objects to be ridiculed.
  • You are never at fault for anything; it is always your stupid boss, stupid co-workers, and stupid clients.
  • There is no possibility of change. Every attempt at change will be thwarted by the system.
  • There is no possibility of escape. Everywhere else you go will be equally bad.

Would you want to be friends with someone who believes these things? Do you want to become a person who believes these things? Then why would you read Dilbert?

As a corollary of the above lessons, here are a couple of plots that you will never see on Dilbert, even though I think they make sense in the context of a dysfunctional workplace, and could be funny.

Plot: Dilbert runs a job interview with a smart, capable young college graduate, who passes all his questions with flying colors. Dilbert takes him aside and whispers in his ear, "Run. Run now, while you still can."

Reasons you will never see this: (1) Dilbert never shows any compassion for another human being. (2) Nobody on Dilbert is ever competent at anything except Dilbert himself. (3) It would raise the uncomfortable question of why Dilbert himself does not leave.

Plot: In a long-running arc, Dilbert leaves his office to found his own startup, small business, or consultancy. He makes all kinds of hilarious mistakes, of the type which founders inevitably make, thus bringing his business to the brink of failure. He finds that he can only be rescued by dogged persistence and the help of mentors and allies.

Reasons you will never see this: (1) Dilbert can never be shown to be at fault for anything. (2) Other people can never be seen as a force for good. (3) The root causes of problems in the workplace can never be shown to arise from the inherent difficulty of making anything work well; it must always be rank stupidity, arrogance, or some other venial human flaw. (4) Scott Adams has no idea how to draw any setting besides a cube farm.

The latter of these is particularly telling, because Adams obviously left his job to found his own business as a cartoonist (and as a result became quite wealthy). But the fantasy he sells to his audience is not one of struggling to change the objectively terrible conditions of their lives, just like he did; it is one of complacently remaining in place while cultivating a smug sense of superiority and alienation. The relationship between Adams and his audience is therefore one of condescension and contempt.

Long ago, Paul Graham ran an ad for his seed capital fund titled Larry and Sergey Won't Respect You In The Morning; Graham was articulating a disdain for large corporate workplaces that is a distant cousin of Adams's, and yet there is a crucial difference. Graham is (admittedly for self-serving reasons) trying to channel his audience's discontent into an urge to follow their dreams and change the world. What would a similar ad for Adams's work look like? Oddly enough, it would be something like this: "I, Scott Adams, won't respect you in the morning. I left my job to build something of my own. Now sit there and read my comics like the powerless peon you are, and were always meant to be. Ha ha!"

Thursday, July 22, 2010

Personal digital curation: a software category that does not exist, but ought to

If you're a typical computer user today, you have lots of data that you've created or participated in creating. The data takes several forms, typically including:

  • Personal documents: writing, drawing, photographs, home movies, etc.
  • Purchased media: music, movies, software, etc.
  • Electronic records: receipts, account records, etc.
  • Communication: email, chat logs, & other social media messages.

There are several things about how you keep this data today which ought to bother you.

First, it's probably inadequately backed up. By this, I mean that you don't back it up frequently enough, and when you do, it's probably either (a) on a USB hard drive or (b) on cloud storage. Taken alone, either of these is inadequate. Typical hard drive backups are inadequate because on most file systems, the data's neither pervasively checksummed nor stored with redundant error-correcting codes; as a result, all your files are subject to random corruption. There's also the small matter that you probably don't archive your backups regularly to a remote location that's safe if, e.g., your home burns down. Cloud storage alone is inadequate because (1) any compromise to your account could result in malicious deletion or corruption of your data and (2) Amazon/Google/... may seem like permanent institutions today, but on the scale of decades I am somewhat dubious; DEC and SGI and Sun were once lords of Silicon Valley; ultimately one must consider Phlebas and all that.

Second, the data's stored in a mishmash of formats, some of which will be exceptionally difficult to read in years to come. Microsoft's file formats are particularly egregious, but I also have my doubts that, for example, today's video file formats, or an iPhoto or Picasa metadata database, will be readable by commonly available software in twenty years.

Third, a lot of your data's stored in multiple related forms, and the relationship between those forms is totally ad hoc and not captured by future-proof software. For example, you might have a batch of raw photos, of which you pick a few to clean up, rescale to lower resolution, and upload to the web. So now you've got multiple versions of the photo. If you need to go trawling through this mess some years from now, you're in for a lot of curatorial tedium reorganizing it and figuring out what's redundant and discardable versus what's a pristine original that you must keep.

Conquering any one of these problems, let alone all of them, requires serious geekery today. For example, if you want to have good backups, you need to store data both in cloud storage and on multiple media, and you need software that records and verifies the checksums of all your files. The other two problems are just as gnarly, if not more so.

This may seem like a totally anorakish concern that doesn't matter to most people. Maybe most people are OK with most of their data being ephemeral, except for the rare object that they print out into physical form. Maybe it's just me, because I've been thinking about posterity a lot lately — including photos and video, my raw data generation rate has risen to something like a couple of GB per month. But I suspect I'm merely one of the people on the leading edge of this problem. Someday, everybody will generate a couple of GB per month and they really will want to share the family albums with their grandchildren without inordinate curatorial effort.

So, as far as I can tell, there's a big gaping hole in the market for personal digital curation software — software that would help you not only back up your data (there's plenty of software out there for that) but that would take care of ensuring the posterity of your data. This implies at least (1) backing it up to multiple distributed locations, (2) transcoding it into future-proof forms, and (3) remembering the relationship between different parts of your data.

This software would not be simple to build. It would have to be cross-platform. To offer a credible promise of future-proofness, it would have to be built on well-documented protocols and file formats so that if your organization went bust, someone else could write software, from scratch, that at least recovers the data. It would have to either include software that manages common file types like photos, or to hook into existing software that manages them, or compute relationships between the files after the fact (for example, it would have to either replace Picasa, or hook into it, or be able to figure out by post hoc analysis when two files were really variants of each other). It would have to be performant. It would need a nice UI.

I suppose the difficulty of building such software is one reason it hasn't been done. Much easier to just build a social networking doodad or a little timewasting mobile app or whatever the next Valley flavor of the month is. On the other hand, I think there's actually a reasonable (although perhaps tough to pitch) business case. There's probably at least tens of thousands of digital obsessives in the world who'd pay Photoshop CS-level prices for a credible digital curation package. The need to support new file formats or cloud storage APIs as they come online could provide a steady stream of upgrade revenue. If you built it right, then there's the potential for standard licensing deals where you bundle value-subtracted versions of the software with new computers, digital cameras, and other doodads.

Oh well. Anyway, add it to the list of stuff that I wish existed but does not, and also the list of things I wish I the time and focus to write but will probably not get to in my lifetime.


UPDATE 2010-08-03: Apparently you can actually learn something by blogging in ignorance and waiting until Reddit sends some commenters your way. There's an IT service category called digital asset management (DAM), and it's a big deal for enterprises (which shouldn't be surprising). (In library science, the analogous problem is called digital curation, which IMO is closer to the problem I care about.) The question, I suppose, is whether DAM can be scaled down, made sufficiently comprehensive, and encapsulated in a mixture of consumer-grade software and services so that individuals can have credible assurance that their data will be preserved on decades-long time scales. I'm somewhat dubious that Expression Media or Bridge can really offer that kind of promise (for example, those packages seem media-focused; do they back up stuff like email and source code?) but of course I haven't looked very deeply. Thanks interwebs!

Wednesday, September 21, 2005

Authors Guild demonstrates technological illiteracy

The Authors Guild is suing Google for attempting to innova--- er, for copyright infringement. If you've been reading this blog for a while, you already know my opinion on the subject. I hereby call for all Authors Guild members either to denounce the actions of their organization, or to stop using all web search engines, or confess to being complete hypocrites. Those are the only options.


UPDATE 22 Sept.: As usual, Lawrence Lessig says it better than I could.

Saturday, October 23, 2004