Economics and similar, for the sleep-deprived

Wednesday, September 05, 2007

 
Freakiology

yes folks it's part 4 of the "Freakonomics" review, now entering into its third glorious year!

Welcome people who came here via direct links! If you are British, would you mind reading this post too? Thanks awfully. If you aren't British, as you were.

Part Four of this review is meant to deal with the odd and slightly patronising relationship that Freakonomics has with normal sociology. I've kind of pussyfooted around this subject so far (although there is a bit about it in the crime gangs section), and include by citation Henry and Kieran (yer actual sociologists) from their contributions to the CT book seminar. I think Henry and Kieran were both wildly too generous in their assessment of the book - you can see that they made a number of caveats and criticisms, but basically welcomed it as being both healthy economics and an interesting contribution to sociology. I take a much more cynical view of both these possibilities - note however that a lot of Henry and Kieran's comments are about Levitt's more general research program and this review is of the specific book "Freakonomics". Let's start off by asking an interesting question - how did the book itself come to be written?

Well I think it happened this way. A publisher found a hole in their autumn schedule where a "big ideas" pop academic book ought to be. They thought that with the right amount of publicity, they could replicate the wildly successful Malcolm Gladwell phenomenon. Normally, the New Yorker is where you'd go to for someone with the right combination of Big Ideas and easy-readin' prose, but they were out of luck. James Surowiecki would have obviously filled this gap just perfectly, but he'd just signed to a different publisher, so they started casting the net a bit wider. Someone had told them that there was this wacky economist guy in Chicago generating Gladwell-style ideas at a rate of knots, but nobody liked the idea of putting a big publicity budget behind an untried writer. So they brought in a safe pair of hands in the form of Steven Dubner, a workmanlike journalist on New York magazine (Edit: Felix reminds me that "New York" magazine is not the same thing as the New York Times magazine, which Dubner actually wrote for, thanks) who was unlikely to set the world on fire on his own, but who could be trusted to translate Levitt's prose into readable English and add enough gee-willikers analogies and anecdotes to convince the airport crowd that they were understanding what they were reading. And thus was a popademic sensation born!

No, hang on, that makes it sound like the whole thing was churned out like a sausage factory. Let's try again.

Well, I think it happened this way. Dubner was a magazine journalist who was doing all right on New York the NYT magazine, but was well below the exalted level of a Malcolm Gladwell or a James Surowiecki. Realising that he was basically nowhere near as talented or intelligent as those guys, and therefore had no chance of coming up with enough ideas himself to write a "The Tipping Point" or a "The Wisdom Of Crowds", but nevertheless wanted to make a load of money off a potboiler book, he decided that half of sumthin' beat all of nuthin and started looking around for someone with a load of ideas. He got commissioned to write the magazine profile of Levitt and bingo, a cash-in popademic sensation was born!

No, no, hang on, that's much too cynical and unpleasant. Let me start again …

Levitt is a Beckerite, a particularly virulent subspecies of economist prevalent at the University of Chicago and specialising in applying economic analysis to other areas of human behaviour. Most other Beckerites haven't made much of an impression on the field, mainly because they are in general really quite nasty people. Levitt had one unusual property, which was that despite being a Beckerite, he wasn't an asshole and was therefore able to create a career by coauthoring Beckerite papers with people who had important skills that he didn't, therefore lifting them far above the run of the mill of Beckerites. For example, he managed to get Ian Ayres to do his calculus for him, John Donohue to do his statistics for him, Suhdir Venkatesh to do his primary research for him and so on. This got him a degree of academic fame, but John Bates Clark medals ain't going to buy the Jacuzzi. One day, however, he met this magazine journalist called Steve Dubner, and realised that he could pull the same trick of coauthoring. Dubner did his job of hyping Levitt's research to a parodic degree, and a popademic sensation was born!

No, no, hang on, that's really mean-spirited and nasty. Let's try one last time.

Dubner was a good writer who had an interest in social and policy issues, who found someone with really interesting ideas but no audience. Levitt was a personable economist who wanted to get a wider public audience for what he considered to be intrinsically interesting research. Together, they thought that an extended version of Dubner's magazine profile might make a really good book. The publisher, mindful of the Gladwell sensation, decided to publicise the hell out of it, and it took off and they both enjoyed a more or less deserved success.

The point here is that you can always tell any one of a dozen stories about anything, and all of them often have an element of truth to them. John Kay even goes further and says that for a lot of important stories, there ain't no single truth (the untruest thing you will ever hear at business school will come from one of your classmates, and it will be the great falsehood "I Was There And It Didn't Happen Like That"). In Freakonomics, Levitt and Dubner have a real habit of always picking a single story, and usually one which reflects really badly on the people involved.

So, for example, sumo wrestlers are "cheating" because wrestlers who need to win a bout to stay in the top league do statistically significantly better when fighting wrestlers for whom the match is a dead rubber. Not "taking sensible steps to minimise the risk of injury". Not "following unwritten social conventions of the sport". Not anything else, but "cheating", and most likely doing so because of bribery and corruption from betting syndicates. Perhaps one day Levitt and Dubner will look at the behaviour of Tour de France cycle riders, and it will just totally blow their minds.

Similarly when it turns out that estate agents' houses typically sell for about 10% more than comparable houses owned by non-estate agents, then this shows that those rascally agents are ripping you off!!1!. That's it! Personally, I don't think 10% is that much. If I went to Aubergine and got a meal 90% as good as Gordon Ramsay gets when he eats there, I'd be pretty damn pleased. Furthermore, estate agents selling their own houses don't have to deal with their horrible emotionally needy clients making all sorts of irrational decisions, and are more likely to be moved by financial considerations than by which buyer was nice to their dog and more likely to keep their gazebo. And so forth, and so on. It's a version of what George Orwell was talking about in bemoaning the habit of self-styled "sophisticated" Marxists to scout around for the lowest and meanest possible motivation for anything and then proclaim it the "real" one. I've often joked that neoclassical economics and evolutionary psychology are unusual among social science fields because they're the only areas where you can gravely insult someone by agreeing with their theory (by telling an economist that he's only saying what he's saying because he's well paid for doing so, or telling an EP type that his latest paper is simply an attempt to draw attention to himself and gain status). Freakonomics is fairly and squarely in that tradition.

I think a large part of the trouble here is that the whole project of attempting to apply the methodology of economics to the kind of behaviour studied by sociologists is that the "rigour" of classical economics comes from a reductionist approach, and for this reason it really doesn't handle "thick" concepts, with non-nugatory social meaning to them, at all well. For example, sumo wrestlers on the brink of relegation do well against superior wrestlers who are sure of staying in their own division. Certainly sounds like evidence of cheating.

On the other hand, as I've noted above, there are a lot of other potential explanations. First, of course somebody is going to fight harder in an important match than in a dead rubber. Second, there are all sorts of unwritten conventions in sports which can be analysed as repeated games, where people pass up potential short term gains to maintain an arrangement which suits the long term interest of all (in cricket, bowlers pass up a lot of Mankading opportunities, for example) and the wrestlers might be following one of these. I actually think that this is the more likely explanation for the pattern of behaviour seen in Levitt's work, as it doesn't make much sense to me for this pattern to be the result of match-fixing by gambling syndicates. If you're carrying out betting coups, you want a match with a clear favourite and with a lot of betting on it, and you do it as infrequently as possible[1]. Fixing nearly every single one of a large category of evenly matched battles between no-marks is about the opposite of economically sensible behaviour[2].

But third, and most importantly, what's "cheating", within the world of sumo? Fixing title fights certainly is (there is a lot of evidence that this happened, although it's tangential to the work in Freakonomics). But how about "not using your best moves on Joe-Bob who has had a bit of bad luck in the tournament, in return for which he will lie down for you on your next match en route to the title"? After all, carry out even a cursory analysis of mountain stage times in the Tour de France and you'll pick up evidence of the "charabanc" - the arrangement whereby the sprint specialists ensure that the gap between the leader and the back of the field does not get too large to disqualify the sprinters. This will certainly show that some riders go much slower on mountain stages than they otherwise would be capable of. But publish a paper demonstrating that you've found evidence of collusion between teams at the back of the pack on Alpe d'Huez and the team from L'Equipe will probably ask if you're someone's simple brother.

There are a hell of a lot of "you scratch my back and I'll scratch yours" arrangements in the world. Some are illegal, some are disreputable, some are more-or-less standard and some are pretty much constitutive of what it is to behave honourably. I think you'd have to know a hell of a lot about the culture of sumo wrestling in order to be 100% sure that this mapped unambiguously onto the word "cheating". I even bet there would be a lot of disagreement between people within that world about the nature of the implicit contracts there. That's what sociologists exist to tell us about, and one is not doing a service to the world of knowledge by pretending that these thick categories can be flattened out. Economics' big claim to being a science is at least partly based on its formalism, but if you try to investigate a lot of social phenomena without paying sufficient attention to what your data points mean then … well, then you're going to end up concluding that a crime gang is more or less like a capitalist firm with an "up or out" promotion structure [3].

But hang on … surely the special status of the claims of Freakonomics is underwritten by the fact that they're driven by the data? After all, there are plenty of bits of research in Freakonomics that don't really fit into the economic paradigm at all - the childrens' names, or the cheating teachers. So maybe we should read through the verbiage about "incentives" this and "optimisation" that, as being sort of vestigial Beckerism that will fall away as we develop a brand new field of "data-driven social analysis" (to quote the CT seminar).

Well I think not. The problem is, which data are driving Freakonomics? The answer certainly appears to be "whatever data happened to pass across Levitt's desk". There is no guarantee that the Micawber approach to data-gathering is a valid one. Sociologists in general ask a hell of a lot more questions about the validity of their data, what exactly they are measuring, what implicit assumptions are made in the data gathering process, etc etc ad nauseam, than economists do. There's a reason for this.

The reason is that economists are in general used to working with standard data that is gathered by official bodies. Twas not always the case. Sorting out what Gross Domestic Product was meant to measure, how to measure it and all the related issues in the system of national accounts was carried out in the 1930s and there is a reason why the third Economics Nobel ever awarded was given to Kuznets for doing it. But for a period of time longer than a generation of economists, the basic material of economics has been hard numbers, collected in annual statistical abstracts, checked, collected and audited by Someone Else. Wasily Leontief was complaining about the atrophying of data-gathering skills in the profession back in the 1950s, and it's got worse since.

But anyway, the point is that "the data" by and large has a fixed referent in the context of economics - it's the same data which everyone else is using, collected by and large for a known purpose, by people without much axe to grind. And also, in economics, the question that the data has been collected to answer is the same as the question you're asking, because economics has intentionally restricted its subject matter to the organisation of production, consumption and exchange. There's no question about other aspects to the problem because economics isn't interested in the non-widget significance of Amalgamated Widgets (or for that matter, in widgets except in as much as they are bought or sold for the numeraire).

You just can't say that about "the data" when the referent is "this dataset that I've just come across", or you're going to make two categories of mistakes. First, you're going to make mistakes of data quality control unless you're very careful, because there's no equivalent of the BLS underwriting the quality of "the data" that you're using. This is a widespread problem in actual economics, by the way - it's genuinely frightening how many otherwise sensible economists use the Freedom House or similar "Indices of Democracy" as if they were useful numerical proxies for the state of social and political institutions, despite the horrendously tendentious way in which those scores are compiled. And even when the data aren't being compiled by someone putting their thumb on the scales, the greater degree of freedom that you have in picking up your "quirky" datasets means that there's much more due diligence required in making sure that you haven't picked up a lemon (and much more self discipline needed to make sure that you're not data-dredging, a subject covered in previous sections).

And second, there's a big danger of mistaking the map for the territory - assuming that you've got the answer, that "the data" tell you, rather than an answer which the combination of the data you happened to pick up and the model you happened to impose on it told you. I said in part one that Levitt & Dubner have a really bad habit of saying:

"Whichever way you look at the numbers, X"

when all they can really justify is:

"Whichever way I look at the numbers, X".

but in fact, I should have said that they could only really support:

"Whichever way I look at these numbers, X".

So there's a big danger of coming up with a partial, misleading explanation, and "the data" will not warn you when this is the case; there's a big danger that you'll end up chasing a lot of charabancs, in the sense of finding explanations of social phenomena as solutions to short-term optimisation problems while ignoring the long-term context in which they are placed. The best you can hope for is that a passing sociologist, or someone with specific institutional knowledge, will set you right (in which case the game theory textbook is certainly big enough for you to come up with a "just so" story fitting the new explanation into your theoretical framework as being "really" an iterated game; the Davies-Folk Theorem applies here). But even this best case is not really adding anything to the sum of human knowledge; it's just an example of one discipline trying to colonise another.

Which brings me back to my fundamental problem with this book - the attempt to simultaneously hunt with the quantitative hounds and run with the qualitative hare. The jacket of the book advertises a universal, general toolbox that gives definitive answers to important questions. It advertises this in a way that, say, the latest Gladwell, or "The Wisdom of Crowds" doesn't. Those books are meant to be loosely themed compilations of the ideas of clever journalists, showing you one way of thinking about some interesting questions, which sometimes works. The big selling point of Freakonomics was that this was meant to be science. This was meant to be the one in which you got the real answers, if you were only smart enough to follow the true path of incentives and data. And it doesn't, because it can't, because good economics is bad sociology.

Quite why this expedition to try and make economics do the work of sociology was first carried out, and what it means for the current state of economics, will be the subject of part Five.

[1] Or alternatively, there is the Malaysian method on cricket, where you fix the betting on something other than the actual result (overs bowled, wides, minutes of play, etc). But this would depend on the existence of a much wider and deeper betting market than the fairly nugatory underground one which exists on sumo.

[2] This is not the first time I've had cause to question the thoroughness with which the "thinking like an economist" model is being applied. Remember in part One of the review, where I was asking why it was that the sales force of that crack gang were, according to the accounts, being paid flat wages with no element of sales commission? More about this in part five, on the general theme of there really being no substitute at all for specific institutional knowledge.

[3[ I think that the crack gang chapter is the worst mistake in the book and the one which really proves that the "Freakonomics" methodology leads to serious errors. It is quite likely that the average sumo fan (although perhaps not the average wrestler) would regard tacit collusion on the final day of a tournament as being cheating. But a crack gang (even a really weird one in which all the members are paid salaries and don't take sales commission, sorry to bang on about this but I mean really) isn't like McDonalds, or like McKinsey, or like any other recognisable capitalist organisation. Proof; even really primitive cultures have groups recognisable as youth gangs, but they don't have groups recognisable as corporations. Again, social meaning. The purpose of a gang is not just making money. Fifty Cent should not be taken as a reliable witness on this issue.
31 comments this item posted by the management 9/05/2007 08:35:00 AM