Economics and similar, for the sleep-deprived
A subtle change has been made to the comments links, so they no longer pop up. Does this in any way help with the problem about comments not appearing on permalinked posts, readers?
Update: seemingly not
Update: Oh yeah!
Wednesday, January 31, 2007
More of the same, if you like that sort of thing
I have a post up at the Guardian blog, which would probably fit reasonably well into the "One Minute MBA" series here, in case anyone cares. I am currently resisting the urge to write something about Nick Cohen's book, but it has that sickening feeling of inevitability to it. The problem is the book itself - I don't really feel comfortable with the idea of reviewing the book without reading it, and I can't read more than a couple of pages at a time. What all Nick's mates call "fantastic polemic" just seems to me to be the equivalent of watching television with the colour, contrast and brightness all turned up to maximum. Julie Burchill's journalism often used to have this effect on me, although I actually find her much more readable these days.
John Harris, in the course of a very sensible article, notes that "What's Left" mentions Robin Cook on two pages while devoting sixteen pages to Gerry Healy of the Workers' Revolutionary Party. As true trainspotters will be aware, this is the result of Oliver Kamm's advice and against Nick's misgivings. Advantage: blogosphere or something.
this item posted by the management 1/31/2007 01:20:00 AM
Tuesday, January 30, 2007
Fangre? Beth yn y blydi hel yw fangre?
Welsh language politics throws this sort of thing up from time to time. Via my brother in Daniel Davieshood, Daniel Davies.
It really is an utterly obscure word. It would be roughly equivalent to the English signs having to say "Smoking in the demysne is prohibited by law". I am surprised, though, that out of all the Welsh bloggers to have posted on this, I am apparently the first to say "Didn't we have a lovely time, the day we went to Fangre".
this item posted by the management 1/30/2007 09:56:00 AM
The Norms of Civilised Debate
Update It's over at the Guardian blog too. I take the opportunity to remind Norm that if he wants to bring on topic the subject of people "ignoring the Iraqis and just wanting to say I told you so", then it is not necessarily going to reflect all that well on him and his mates.
Oh look. Norman Geras is having a go at me and Marc Mulholland. It's in this post, in the otherwise incomprehensible third paragraph on the "clever-clever approach". Norm has been ... reluctant to mention me by name for quite a while now, and he appears to be giving Marc the same treatment now. I can't speak for anyone else, but I am certainly weeping bitter tears over the snub. What a jolly grown up way for the Professor Emeritus of Government at the University of Manchester to behave.[1]
Given that he has had no less that nine months to gestate this one, (counting from the day I first made the point), it's not much of a show for the wait. The "clever clever" argument isn't "clever-clever" at all - it follows from the simple meaning of the words "pro" and "war", and Norman Geras has never responded to the actual argument made. But even if we take "pro-war" to be restricted to "pro the war in Iraq", I respond thus.
1) The Euston Manifesto says, twice that one of the things it is against, is anyone who thinks that the decision to go to war in Iraq should have political consequences for the people who made it. Specifically, it says once that the authors are not interested in "picking over the rubble"[2] of the intervention, and once that the authors "have no truck" with people who spend what the authors perceive to be too much time and energy on insisting on political consequences for the decision to fight a war in Iraq.
2) The Euston Manifesto does say that horrible regimes should be subject to intervention, and it does say that Iraq was such a regime.
3) Whatever the main drafters thought in 2003, as far as I can tell all of them are currently in favour of coalition troops remaining in Iraq, none of them have expressed an opinion against the US policy of increasing troop numbers in Iraq and as far as I can tell all of them were in favour of continuing to occupy Iraq at the time they wrote the Euston Manifesto. The war in Iraq is not actually over, whatever George Bush said about "major combat operations".
4) The simple fact that someone opposed the war in 2003 does not make them "anti war" for the rest of their life. Nick Cohen was against the liberation of Afghanistan at the time, but presumably Norm would not call him "an opponent of the war against the Taliban". Or maybe he would, I have no idea. I have had a bit of a look and as far as I can tell, all of the "anti war" members of the Euston Manifesto committee refer to their being "anti-war" in the past tense; "I was anti-war" rather than "I am anti-war".
5) That cricket analogy is just totally fucking meaningless[3]
So in other words, the document says that the Iraq War was the kind of war that left-wing people should presumptively support, that left wing people should not expend political energy in criticising the decision to fight the Iraq War, and that the troops currently occupying Iraq should stay there. To borrow a phrase, I have no truck with the tendency to pay lip service to the anti-war case, while devoting most of their energy to criticism of political opponents at home (supposedly responsible for every difficulty in Iraq), and observing a tactful silence or near silence about the lies told and the disastrous execution.
In other words, Eustonites, you're the pro war left. Everybody knows it. You're not fooling anybody. Wear it.
[1]Marc had the worst of this, as Norman wrote this post in which he said
"Why, I even used to discuss things with Marc Mulholland, until his blogging turned obsessive in a particular way that I've come to know and leave alone.".
I didn't speak up at the time, but in retrospect I think I should have and now I think I will. This was, in my opinion, utterly unfair in that Marc had not in fact pursued him more than twice for a straight answer to the straight question that he is only now answering, really quite disgusting in the slippery implied accusation that Marc was behaving like a loony, and a really bad example of the kind of behaviour that gets blogs a bad name. It is also really not fair dealing to tell your readers that you are addressing an argument made by "some people" without saying who they are, or providing a link so that people can see for themselves whether you're addressing a strawman.
.
[2] Yes, "picking over the rubble" is what the Euston Manifesto says about Iraq, and whoever came up with that phrase ought to be made to wear it like a crown of thorns until the day that the last bombed building in Iraq is rebuilt.
[3] Since I am a uniter rather than a divider, I will point out to "some people" (by which I mean Norman Geras, although I have decided to become too snooty to say so) that if your main audience is American, it might be a good idea to cut down on these cricketing analogies, as in my experience, Yanks tend to find them irritating rather than cute.
this item posted by the management 1/30/2007 09:09:00 AM
Friday, January 19, 2007
DW Randall, run out, 13
Shorter version: We, bloggers, are the people. That's why we're so horrible, and that's why we will never have any influence. (this joke stolen from "The Poor Man")
I am obviously left on the sidelines of the current War o' the Blogs raging between "Guido Fawkes" and "Bloggerheads". I don't link to either of them anyway on my tiny little link list, and all of the people I do link to are mates of one kind or another and therefore not susceptible to de-linking more or less whatever they do. In general, I am not a big fan of ostentatious delinking out of nothing more tangible than a vague sense of it being a bit knobby, but I thought I'd consider this campaign a bit more carefully as it gives me an opportunity to have a go at the progressives again, sort of.
Tim Ireland, proprietor of "Bloggerheads", has a number of worryingly progressive tendencies, specifically a tendency to regard public participation in the bizarre minority hobby of politics as an end in itself, and a little bit of a lean towards self-superiority in his attitude to people who don't regard his projects as politically interesting. And he's about 36% more hung up on the importance of "the media" than I personally regard it as sane to be. But in general, I more or less regard him as a good bloke, mainly because his progressivist projects are directed toward somewhere where they might be of use. He expends his energies for the most part on having a go at professional or semi-professional politicians, telling them how to go about their lives, rather than the more common progressive project of doing the same to the man in the street (or just as bad, to working journalists, who have to earn a crust like the rest of us). Since politicians have by definition accepted that this is an OK thing to do, I regard Tim's projects as no-harm, no-foul, and if his utopia of blogging politicos does come about, at least I can have a pop at them in the comments section.
Guido Fawkes, on the other hand, has the stated ambition of being the British Matt Drudge. This is actually probably a realistic ambition as far as I am concerned; back in the 1990s, I used to read the Drudge site quite avidly, then gradually realised that all the stuff on it was actually extraordinarily dull, and until two minutes ago when I checked it was still there, I probably hadn't thought about it for five years. I am in roughly the same state with the Guido Fawkes blog. The trick that those sites have (and a large proportion of Private Eye too) is that they appeal to your vanity. They present the information in a conspiratorial "not everybody knows this" kind of way, so you look at the story and think "aaahhhhhh, so that's what's going on, actually I do seem to remember hearing about that, I must be an uncommonly sharp and switched-on individual" rather than the more logical response of "I read that in the paper two weeks ago". Every now and then they get a little bit of a scoop, but it is almost always of the variety "hey, have you heard? Bob called Terry a cunt! I wonder what Thelma will make of that?!?". Stuff that might or might not be true, but it hardly matters because in five minutes you will have forgotten about it, because a) it is very rare that you remember who Bob and Terry are, and b) even if something important does happen as a result, you will never connect it to the original story because Thelma will not provide a footnote saying "actually, it was all because Bob called Terry a cunt".
I kind of want to think that Guido is a man after my own heart though, because he does hate all politicians and by and large, so do I, so we're both negativists together. But when you look at that site, I don't think it really actually qualifies as a negativist blog at all.
Recall from the original "shit on progressives" post, that my argument is that I have no particular reason to believe I have any talent at all in suggesting policies or schemes. Therefore, the only way that I can participate in politics for the most part, is to criticise the flaws in the schemes proposed by others. Since the current government proliferates schemes with large and dangerous flaws at such a rate as to completely exhaust the time and effort I am willing to spare on this hobby, a rule of thumb would suggest that my time allocation should be 100% to criticising politicians and 0% to anything else. So my rule is "negative comment only".
But that isn't the same as "negative comment always". The point is that for any political thing you are expressing an opinion on, you have the option of 1) give negative comment, 2) give positive comment 3) give no comment. My thesis of negativism is that 2) is, apart from a few oddball cases, almost always a waste of time, since government projects don't need support from me, and non-government projects are unlikely enough to succeed that a version of the Voter Paradox kicks in. Effort is better expended on projects of type 1), because the probability of being the equivalent to the "marginal voter" is much greater - it's much, much easier for a mass movement to stop something than for a mass movement to start something. Of course, this means that fewer things get started, but I think that I covered that bit in the original progressivism post.[1]
So, from me, you will get negative comment, or nothing[2]. "Nothing", can be interpreted as an integral over the range "sullen acquiescence : enthusiasm". The point is that there is some differentiation between things I regard as dangerous ideas which ought to be opposed, and things that aren't, and this is my gesture of good faith, to show that I am actually taking these things seriously. If I issued a boilerplate statement of excoriation on everything that crossed my in-box, then this would be something different. Just as, my general hatred of Hazel Blears and John Reid has to be juxtaposed with a degree of regard for Hilary Benn and Gordon Brown (and even Alan Johnson, the minister), because if you don't make an honest assessment to assess the competence and honesty of politicians, then you're indistinguishable from a nut and/or idelogue.
This is what I don't like about the Guido Fawkes blog. It's a non-indicator. The author just shoots at every target that pops up, like one of the recurring jokes in "Police Academy". And he's doing it all in order to promote a wider project; a specific, libertarian-in-the-pejorative-sense small government agenda.
That's just more progressivism, with a minus sign in front of it. Constantly giving politicians the benefit of the doubt, in service of the cause of encouraging us all to put up with an encroachment of the political sphere into the rest of our lives, is in my opinion bullshit, but constantly giving the politicians stick, in the cause of encouraging us to put up with a roll back of the political sphere from parts of its proper territory, is no less bullshit. So on careful analysis, I don't consider myself part of that particular strain of politics.
So anyway, I am on the whole not up for a massive blogular boycott, of "The Sun in Liverpool" proportions, but I think that Tim Ireland's post has reminded me that the Guido Fawkes blog is correctly occupying that part of my mind marked "blogs that cause me to lose a bit of respect for people's opinions if I discover that they're really keen on them". Along there with Little Green Footballs and Daily Kos[3]. So if you wanted my opinion, you now have it.
PS: They do this sort of thing so much bigger and better over in the states.
PPS: I am in substantial agreement with Tim Ireland on everything he says about ethical comments section management policies, although to find out whether Guido Fawkes is actually guilty of the specific crimes concerned sounds like more work than I care to do. To be honest, if society has reached the point at which blog comments sections have any influence on anything important, then we are well into book 5 of "Decline and Fall" and the only thing to do is start moving things around a bit to make it more convenient for the cockroaches when they take over.
Conflict of interest declaration: I don't think I've ever communicated with either of them in any way at all. Guido Fawkes did copy me in on a mass email with a fucking great Flash file attached to it around the time of the last election, which I seem to remember irked me a little bit at the time - I'm not going to categorically accuse him of having spammed me, but I certainly don't remember ever having asked for it.
[1] Specifically, I am not in favour of any wholesale reorganisation of politics aimed at making it easier for groups of ordinary citizens to get together and "make things happen" in terms of big or even medium-sized projects of the kind currently carried out by government. The very idea gives me the shivers, frankly. I am of the unfashionable (is it? I don't have any idea what's fashionable these days) idea that parliamentary democracy basically works, in the sense of assembling people into administrative tasks who are more or less able to do them. I have no great belief in the ability of small largely self-appointed groups to do a better job. And (in a phrase that intelligent readers will surely see through as pure bluff) I think that, properly read, Hayek agrees with me.
[2] Obviously I don't stick to this as a hard and fast rule in every single case, what kind of a madman do you think I am?
[3] Just to be clear, I thought that the Daily Kos was a nest of bores and nutters long before I received any death threats at all from them.
this item posted by the management 1/19/2007 07:31:00 AM
Tuesday, January 09, 2007
This has been so absurdly trailed it is bound to be a total anticlimax
yes folks its
The long awaited Freakonomics review Part 3:
Happy New Year to most of my readers. Time to crack on with the Freakonomics review, which I actually wrote last year, but wanted to hold back until I'd finished the whole thing (yes I know, touching isn't it?). Thanks to Radek for giving me the heads up on this review, which makes a number of good points. The remaining parts are tentatively entitled "Freakiology", on the subject of Levitt & Dubner's in my opinion very sketchy treatment of important issues of sociology, and "How Freaked is Economics?", on what the success of Freakonomics as a popular book, and the success of Levitt as an academic economist, say about the current state of the science of economics. This bit rounds off the statistical and methodological critique. It is entitled ...
Natural experiments ain't so Freaking Natural
This section of the Freakonomics review deals with my problems with the underlying econometric methodology of Freakonomics (or more specifically, with themes in the career of Steven Levitt which are summarised in the book). It expands on this post from a year ago, in which I got rather alarmed at Levitt's reaction to being criticised in a working paper on his abortion & crime model. I need to start with a big caveat; if you're criticising someone's statistics, it is easy to get into the realm of the unprofessional and/or defamatory. Nothing I say below should be taken as accusing Levitt or any of his coauthors of intentionally misrepresenting anything. In particular, references to "data-mining" refer to what I regard as a deformation of the general field econometric methodology rather than to anything purposeful or specific to Levitt. So here we go.
Levitt's reputation in economics rests on micro-economic empirical work rather than theory. Empirical microeconomics is in general a rather hairy mathematical field; unlike empirical macroeconomics, when you are generally dealing with a relatively small number of well-known and consistently collected aggregate time series, microeconomic datasets tend to be idiosyncratic, problem-specific, collected in ways which might be considered to introduce bias, qualitative (ie, yes/no) rather than quantitative and statistically hairy in all sorts of other ways.
This is why microeconometricians tend to be much stronger on nonlinear methods than macroeconometricians (although generally weaker on time series modelling) and generally more familiar with the dusty end of the STATA instruction manual. Very few macro models are particularly complicated from a mathematical standpoint; they are often big, which makes them complicated in a different way, and in general a lot of thought has to go in to the process of deciding what variables to include and exclude, but the actual statistical guts of the thing is normally a linear regression – probably one that is estimated by maximum likelihood because of the time series issues, but basically a model where the output is a linear function of the input, with parameters chosen to minimise squared error.
Microeconometric models are much gnarlier, almost always estimated by ML (or these days, as often as not, by Bayesian methods which allow for even more complicated functional forms), with structures that are highly non-linear. On the other hand, my assessment of microeconometric modelling is that they don't spend anything like as much time and effort on the modelling issues (as opposed to estimation issues) as the macro guys, and they might be surprised what they found if they did.
Levitt, on the other hand, does a lot of microeconometrics, but he is not a good econometrician (as he freely admits). How does he manage it? Well, partly by having good co-authors (I will return to this issue in part 5). But partly by making very extensive use indeed of the instrumental variables approach.
As part of my "mission to explain", I should probably now explain what the IV approach is and why anyone might be interested in it. OK here goes then. Imagine you are the chancellor of Oxford University, trying to find out whether rugby players are thicker than rowers[1]. You might want to carry out a regression analysis of the form:
Alpha (% of rugby players) + Beta (% of rowers) = Average Finals mark.+/- an error term
, across the colleges, and have a look at the coefficients on rugby and rowing.
However, the dean of your medical school points out to you that this regression won't work. The variation across the finals marks of different colleges is also affected by the amount of beer the students drink. Rugby players drink more beer than the average student, so colleges with a high proportion of rugby players will have lower marks than average, not because rugby players are morons, but because they are also boorish drunks. Also, because you're not on speaking terms with any of the bursars, you can't get any college-level data for beer consumption. Hmmm.
Inspiration strikes. You realise that Welsh students are more likely to be rugby players than the average student, but no more likely than the average student to be a drunk. Furthermore, from a previous year's study you have data on the number of closeted gay men in each college, and it too is well-correlated with the proportion of rugby players. Bingo zingo, it turns out that the college-level data on purchases of annoying novelty hats also correlates well with the rugby players, while the readership of the Financial Times correlates strongly and negatively. So you can estimate a preliminary regression thus:
Rho (% of Welsh) + Tau (% closet gays) + Theta (novelty hats) + Mu (FT readers) = %Rugby players +/- an error term.
Call the left hand side of this equation Gamma. Gamma is a pretty good estimate of the number of rugby players, and (because Welshness, closeted gayness, novelty hat purchase and FT readership are none of them correlated with beer drinking), unlike the raw data for the number of rugby players, it isn't correlated with the variance in the error term for finals marks. You can therefore substitute Gamma for the % of rugby players in the first equation, and your estimates will now be consistent, because you've got rid of the confounding factor of beer consumption. Gamma is an "instrument" for the number of rugby players, and the version of your regression equation which substitutes Gamma for the percentage of rugby players is the "instrumental variable" estimate of the relationship between rugby, rowing and finals marks.
That's IV estimation[2]. Levitt does a hell of a lot of it. As long as the left hand variables of your preliminary regression aren't themselves correlated with the variance in the original equation (in other words, as long as Welshness isn't itself correlated with drunkenness), and as long as the fit of the equation estimating Gamma is reasonably good, it will be OK. (You can actually make do even with a really bad fit in the Gamma equation if you have loads and loads of data[3], but usually you don't). It's a good method of estimating these models, so why doesn't everybody do it?
Well, in the real world (a place I have often visited), you aren't allowed to randomly pluck series out of the air and say that they are strongly correlated with the variable you want them to be instruments for. If it turns out that they are weakly correlated, then you are in hell. The reason for this is that, although we said that Gamma was "uncorrelated" with the error term in the finals marks equation, in any real (finite) dataset the measured correlation is likely to be a small number close to zero rather than the actual number zero. This matters like hell because:
1) the bias introduced by this small empirical correlation gets "scaled up" by the reciprocal of the covariance between the instrument and its target variable (this is now technical as hell, but here's the best discussion I can find). The idea here is that you are trying to explain the (signal + noise) in finals marks with (signal + noise) in the instrument. If there is only a small correlation between the two signals, then there had better be no correlation at all between the two noise terms, or you are just fitting noise to noise and your overall signal/noise ratio will go through the floor.
2) in finite samples, the sampling distribution of the IV estimate is the ratio of two normally distributed variables. The ratio of two normals is a surprisingly complicated distribution; basically, if it is important to you to estimate something which is the ratio of two normals, then you had better hope that the correlation between them is pretty high because as it goes to zero the ratio of two normals becomes a Cauchy distribution, which is in statistical terms "a really awkward bastard to deal with"[4].
Do you see why this took so bloody long to write, by the way? So the take-away here is that weak instruments in IV estimation are really bad news, much much worse than poorly correlated regressors in normal regression analysis. It can actually be better from a mean-squared error point of view to just ignore the bias and do the ordinary regression, if the only instruments you can find are weak. This is important.
In general, in those of Levitt's published papers that I've read, there really is not very much discussion of the strength of the instruments. There is also a hell of a tendency to say that "there is no reason to believe that this is correlated", with a bit of a lacuna where the bit ought to be where you check that it is actually uncorrelated, or to have a look at how any small correlation might get inflated by a weakish instrument. What was that Malcolm Gladwell quote again?
Steve Levitt has the most interesting mind in America, and reading Freakonomics is like going for a leisurely walk with him on a sunny summer day as he waves his fingers in the air and turns everything you once thought to be true inside out.
Yup, always with the waving of the fingers. We should have got suspicious the moment that anyone told us that econometrics could be fun. In fact, as with all statistical work, the ratio of inspiration and creativity to meaningless grind is so low that it is scarcely possible to reject the null hypothesis of no fun at all. In fairness, a lot of the work that made Levitt famous predates a lot of the weak instruments literature - it is only comparatively recently that it has even become standard practice to report the results of the first-stage regressions so that everyone can make their own mind up about the strength of the instrument. And Levitt is actually quite good by the standards of econometricians when it comes to doing crosschecks and similar non-data-driven tests of whether the model is working or not, which is really the only "solution" to a weak instruments problem at present (people keep working on statistical refinements and there are a few goodish rules of thumb, but basically weak instruments is an unsolved problem of estimation theory). So it's not that this is an awful thing about Levitt; the point I want to make here is that the idea that creativity and flair can substitute for the hard yards in econometrics sounds like a free lunch and it probably is.
On the other hand, however, in most cases, it looks to me as if Levitt is using quite strong instruments (the seminal abortion 'n' crime paper is an exception though; without doing the work, it looks to me as if the crack epidemic in the data takes nearly all the strength out of the instrument Levitt & Donohue were using). I didn't really want to write this piece about weakness of instruments. The real critique I have is based on the way the instruments get found.
Levitt is a hell of a one for "natural experiments". A "natural experiment" is a subspecies of instrumental variable estimation, taking advantage of some natural or otherwise exogenous variation to create a situation where some units get assigned to a treatment group and some get assigned to a control group, by chance rather than design. The stereotypic textbook example is one where you want to investigate whether military service creates human capital (whether ex-soldiers do better in civilian life than non ex-soldiers), but you think that there might be some unobserved characteristics (like self-discipline or bravery) which affect both the decision to sign up, and later success. So what you do is take the cohort of men born in the early 1950s, and use their draft number as an instrument.
Natural experiments are another of those "why doesn't everyone do econometrics this way?!?!?" areas. The answer is twofold. The first part of the answer is that natural experiments are really quite hard to find. Things like the Vietnam draft don't really come along with anything like the frequency at which econometric problems arrive which look like they'd be amenable to an econometric approach.
Levitt's big thing, the one that won him the John Bates Clark and the fawning adulation of millions of groupies Steven Dubner, is being really creative and unconventional in the selection of quirky things which can be used as natural experiments. This is the whole selling point of Freakonomics - it's all about this sort of lateral thinking and "making you see the world in a whole new light".
Which brings me to the second part of the answer which is, unfortunately, that since the success of Freakonomics, every bugger does use natural experiments, all the time. Levitt's book is Edward de Bono for the green eyeshades set. I am wholly suspicious of this outpouring of creativity on the part of economists, rather as I would suspect and fear a sudden outbreak of interest in stochastic calculus among teachers of modern dance.
The problem is that the upsurge in economists finding natural experiments is not a result of there being more natural experiments to find, but a result of economists deeming more things to be acceptable natural experiments. This is worrisome, from a statistical point of view, and it is here that the discussion of "data mining" shall begin, so I redirect your attention to the disclaimer above in which I make it clear that I use the phrase in a sense which is pejorative from a methodological point of view but not personally
The trouble is that there are two ways in which you can go about discovering a natural experiment if you don't have an obvious one to hand. You can either be more assiduous in searching for them, or you can lower your standards as to what constitutes a decent natural test of your thesis. Of these two, oddly enough, I regard the second as much less potentially harmful. It just gives us a social phenomenon whereby not a robin can fall without some lazy graduate student or junior faculty member using its passing as a "natural experiment" on the market for bird seed. It tends to mean that crap papers proliferate in the journals (in general, purporting to prove propositions that nobody was ever disposed to doubt, using econometric techniques so bad as to make you doubt it after all), but it is hard to get worked up about this on opportunity cost grounds, as the authors of these papers would be churning out crap of some kind or another anyway.
The first phenomenon, however, is more subtly pernicious. Choosing natural experiments is a form of data-mining. Since all sorts of things are happening all the time, if you are prepared to get really creative about it, and prepared to put up with weakish instruments in an IV estimate, you are often able to find all sorts of natural experiments for propositions of interest if you look hard enough. Specifically, you will as likely as not be able to find one which gives you the result you are looking for.
I direct readers now to my discussion of data mining and stepwise regression from a couple of years ago. And to this stupid joke from roughly the same period, in order to point out that the decline in quality of this blog since then is largely illusory. The point I want to make is that the natural experiment version of data-mining causes just the same problems as stepwise regression.
Recall that in the case of stepwise regression, it became impossible to interpret the normal tests of statistical significance, because the critical values of the test statistics assumed that the underlying process was a random one. And the process which generated the test statistics wasn't a random one, because it had been specifically set up to iterate through combinations of regressors until a model was found with the "right" result.
I think something exactly similar could be at work in the natural experiments literature. We just don't know how many potential "natural experiments" were looked at and didn't work out, and why. In many ways, we're even worse off than we were in the stepwise regression case, because there is at least a sensible mathematical way of getting an idea of the size and shape of the space of possible regression models that a data-miner has iterated over, and constructing an algorithm like PcGets in order to do so in as sensible a manner as possible. There is no such objective way of dealing with the potential space of natural experiments.
I note here that, as with the stepwise case, the point has to be made that simply trusting in the honesty of our econometricians isn't going to do any good. As I pointed out back then, the double blind criterion is not used in medical tests in order to protect us from dishonest experimenters. It's there to protect us from unconscious bias, wishful thinking and the temptation to find rationalisations for a course of action that is most congenial. And as far as I can see, there is simply no way to introduce any equivalent of the double blind into this form of econometrics.
Another way of describing this problem is to notice that the business of coming up with a natural experiment to test some hypothesis or other is basically the same thing as looking for a piquant anecdote to illustrate a point. It's the same sort of thing that Gladwell or Friedman do, without the statistical manipulation. And to be honest, the econometric toolkit does not actually add anything much at all to the evidentiary value of a natural experiment - all the persuasive power is in the selection of the "experiment" itself. I think that this is both a bad thing about the natural experiment literature and a good thing about anecdotal evidence and case studies (which are, at the end of the day, often a good way of backing up a hypothesis about the world). There is nothing wrong with what Gladwell does, but it is a mistake to think that one is adding anything by taking the semi-attached anecdote and turning it into a regression. Or to put it another way, the plural of "anecdote" is not "data" - it's "Freakonomics".
Postscript I think it makes sense to repeat a third time my disclaimer above that I am specifically not accusing Levitt of sharp statistical practice. My dislike of the natural experiment methodology is general, and while Levitt is the poster boy for its renaissance in American economics, I think he is simply the expression of much wider trends, which result from much deeper pathologies of the subject, which I'll be dealing with in Part 5. Note in particular that I've poured a lot of scorn in the past on the "Devastating Critique" school of statistical rhetoric as exemplified by Steve Milloy, where one takes an utterly standard limitation of some methodology or other (canonically, a suggestion for further research made in the paper itself) and inflates it into a "Devastating Critique" of the methodology itself. I haven't changed my mind about "Devastating Critiques" and didn't intend to deliver one here myself. A fair old amount of Levitt's work does not use natural experiments, and not all of his natural experiment work is necessarily data-mined. But a lot of the key claims made in Freakonomics look to me to be based on "Just So" stories where opposing "Just So" stories could easily be told (Ariel Rubinstein's review picks out a lot of them), and Dubner does not seem to realise how impressive and definitive it isn't that Levitt has converted his "Just So" story into a model.
Parts 4 and 5 to come some time between next week and the heat death of the universe.
[1] Strictly speaking this regression wouldn't answer that specific question but cut me some bloody slack here will you.
[2] Specifically it's "Two-stage Least Squares". IV is a bit more general than this; it is also possible to do it through the General Method of Moments and Limited Information Maximum Likelihood, which I am fucked if I'm going to explain because I barely understand them myself. However, Levitt often uses 2SLS, and I think most of my comments here also to GMM and LIML estimation too.
[3] Everyone says this but it isn't really true. Asymptotically, the IV instrument is unbiased, which is what I mean here. But weak instrument bias can be a problem in finite sample estimation even with huge numbers of data points - famously, Bound, Jaeger and Baker found it in a study with 329,000 data points
[4] Benoit Mandelbrot advocates the widespread use of the Cauchy distribution for capturing the uncertainty of a wide variety of modelling situations. Oddly enough, many past colleagues describe Benoit Mandelbrot as "a really awkward bastard to deal with", which perhaps tells us something about self-similarity in general.
this item posted by the management 1/09/2007 03:50:00 PM
|