Economics and similar, for the sleep-deprived

A subtle change has been made to the comments links, so they no longer pop up. Does this in any way help with the problem about comments not appearing on permalinked posts, readers?

Update: seemingly not

Update: Oh yeah!

Friday, December 09, 2005

If this is normal science I want my old job back

This also "not part of the proper Steven Levitt series", although it anticipates a couple of things from Part 3. It's basically an expanded version of this comment on the Brad DeLong site. Basically, I do not at all like what Levitt is doing on the issue of the challenge to the "More Abortions, Less Crime" thesis in chapter 4 of Freakonomics, and the controversy which has erupted over it since the Foote and Goetz working paper discussed below.

Basically, Levitt has a response up now. He goes hands-up on the programming error, but fights back by making a number of corrections to the underlying abortion data series and says that the key result is still there if you make these corrections. Brad Delong describes this as "normal science", presumably in the Kuhnian sense, but I think it's something a bit worse. Yes, thanks, I am aware that by entering the lists in this way I have become an ally of Steven Sailer, the noted film critic and race nut, but there you go. Here are my specific comments in ascending order of seriousness:

1. I think that the decision to use an instrumental variables approach to allow for measurement error in the Alan Gutmacher Institute abortion survey data is possibly wrong and underjustified (I had to look this up because my recollection of the way you use IV to deal with measurement error is just pathetic; thanks to Ragout in Tim Lambert's comments for helping me out. God this is turning into an incestuous blog project, it reminds me of the Lancet study community). The issue is that if there are measurement errors in the AG data, then the residuals in the regression will be correlated with one of the regressors (because the left hand side is determined by the true relationship, so big residuals will be correlated with big measurement errors on the right-hand side variable). This tends to bias down your estimates of the regression coefficients, so making you more likely to find things not to be significant when they are.[1].

On the other hand, if you're using IV estimation (whereby you replace the series with measurement error by a proxy constructed from other series; this tends to inflate the residuals but takes away the correlation with the right hand side), then the series you use to construct your instrument mustn't be themselves correlated with the measurement error on the original series. If they are, then you're going to introduce a negative correlation and your will tend to bias your estimates in the opposite direction, making you more likely to find things to be significant when they are not.[2]. It might be the case that there is good reason to believe that Levitt's proxy for the AG data from a similar series compiled by the CDC has measurement errors which are not correlated with the measurement errors on the AG data but this ought to have been discussed. I don't like it when people bring in IV estimation with no discussion of why they're sure that their instrument is valid. This is a venial rather than a mortal sin, but Levitt has a chronic case of it; a lot of his work seems to rely on a kind of "gee whiz what an original idea" when coming up with off-the-wall ideas to find measurable proxies for things, rather than explaining in detail why the proxy is valid.

2. On a simple point of fact, the fourth column of row three of the table displaying Levitt's revised results does not show a significant effect. This is the column using the correctly programmed interaction effects and IV estimation (I think it's also using the processed data series but it might not be), so in a sense it's the end of the "improvement" process that's been carried out. This isn't mentioned in the text summarizing the table; again a presentational matter rather than anything else, but irksome.

3. Finally and most importantly, this is about as far from a double blind trial as you can get. I've written in the past about the perils of data mining in econometrics, and to be honest, all that is lacking in the series of changes to the data and the model that the Freakonomics blog presents is a phalanx of dwarves singing "Hi Ho, Hi Ho, It's Off To Data-Mine We Go". What has happened here is that Levitt and his research assistant have sat down in the knowledge that a perturbation to their model doesn't deliver their result, and decided to have a think about what kinds of alterations to the data ought to be made.

You don't need to suggest any intentional dishonesty to say that it is somewhat unsurprising that the outcome of the brainstorming session on "What sort of changes ought one to make to this data, in an ideal world?" was a dataset and model in which the result that Levitt is famous for was present. Even if Levitt and Ethan Lieber had sat down at a table with no computer on it, starting with a blank sheet to discuss the changes to make and not touching the model until they had finished, I would still guess that it would be the easiest thing in the world for someone who was intimately familiar with the dataset to subconsciously put his thumb on the scales. And I don't think this is what they did; colour me cynical but I would bet quids that lots and lots of iterations of different possible changes to the data were tried. I note once more that there is no accusation of intentionally cooking the books here; medical science certainly doesn't insist on double blind trials to protect them from unscrupulous doctors.

I think that there's a general issue here which is endemic to the territory that Levitt chooses to operate in. By their nature, political debates are debates. One side produces arguments, the other side produces counterarguments and so on, so iteratively. This is an environment which is absolutely poisonous to datasets. By the time you've been through two or three iterations of a "controversy" like this it's more or less impossible to pick a model without failing even the most homeopathically weak version imaginable of a double blind criterion. This is why I now say that we're simply never going to know the truth (by which I mean, even the simple statistical truth about the existence of a comovement, much less the truth about the underlying causal hypothesis) about abortion and crime in the period 1976-2000. Stick a fork in this dataset, it's done.

I think it's bad for economics and statistics as a science to start acquiring the habits of thought that are prevalent in these debates (more, much more, on this in the long awaited Part Three). I also think it's bad for politics to have one side of any debate trying to give their case the imprimateur of objective science in exactly the way that Freakonomics does all the time with its "morality is concerned with what should be the case; economics is concerned with what actually is the case" schtick[3]. When your response to a measured, polite working paper is to nip off to the data mines with your research assistant and write a blog post entitled "Back to the drawing board for our latest critics�and also the Wall Street Journal and (Oops!) the Economist.", then what you're doing isn't "normal science". It's normal politics.

[1] By the way, there is nothing wrong with the English phrases "making you more likely to find things not to be significant than they are" or "making you more likely to find things to be significant when they are not", so can we just give up on trying to remember which is a Type I and which is a Type II error, pretty please?
[2] As I say, my recollection of this stuff is terrible. When you do IV estimation in econometrics you are usually doing so because of a different problem (endogeneity) and I have never in my life taken seriously the possibility of measurement error in the series I was using (professional deformation). What I am saying here is that this bit might be wrong, I am staking about a farthing's worth of credibility on it being right and if you want to correct me go for it in the comments.
[3] By this I mean something quite different from claiming that political debates shouldn't be informed by scientific facts, or that social scientists shouldn't get involved in doing work on important topics that they care about; my defence of the Lancet Report on Iraq would look pretty odd if this was what I believed. It's something close to the Humean point about reason being the slave of the passions; scientists ought to be (when they are acting as scientists that is; they can do what they like when they take part in the debate as citizens) simply bringing their best estimate of the facts of the case before the people who decide what to do about those facts. And scientists shouldn't data-dredge, either, of course.
0 comments this item posted by the management 12/09/2005 09:30:00 AM

This page is powered by Blogger. Isn't yours?


Bitch : Lab
Aaronovitch Watch
Brad Delong
The Robert Vienneau blog

Political and philosophical heroes

Subcomandante Marcos
Will Rogers
Boris Vian
The English Svejk

RSS Feed:
This seems to matter to a lot of people

If you liked this "Daniel Davies" website, you might be interested in

"Danux", the web developer
The martial artist (and fan of extremely annoying Flash intros) from Blackburn
The Welsh political journalist
A Scouse computer programmer who collects Soviet cameras
"Danimal", the heavy metal drummer
Canada's finest recorder of radio jingles
More of the same, at the Guardian
A tailor's in Lampeter where Jimmy Carter once bought a hat
An advertising man who has written a novel about dogging (I think we sometimes get each other's email)
An award-winning facilities manager in Dubai
The son of the guitarist from the Kinks Update: he is apparently "balls-out motherfucking shit-dicked exxxstatic" to be included on a Kerrang magazine giveaway CD of Iron Maiden covers, which is nice.
"Fritz Gretel" from the Ramones film "Rock 'n' Roll High School"
The former presenter of the leading politics talk radio show on the Isle of Man, now a business change manager in the Manx government secretary's office
An aquarium curator in Sussex who keeps on scoring home runs like this (this is the first stable link I've found, but he is constantly kicking ass in acquarial terms)

If you didn't like this "Daniel Davies" website, then don't give up on the Daniel Davies industry completely!

An American "Christian Political Analyst" who has the same name as me
A student at Patrick Henry College
these two might be the same guy ...
"Scatter", the deceased Liberian gangster
A naked man stuck in a chimney in Wigan
A thug in Barrow

This blog has been going downhill since ...

August 2002
September 2002
October 2002
November 2002
December 2002
January 2003
February 2003
March 2003
April 2003
May 2003
June 2003
July 2003
August 2003
September 2003
November 2003
December 2003
March 2004
April 2004
May 2004
May 2005
June 2005
July 2005
August 2005
September 2005
October 2005
November 2005
December 2005
January 2006
February 2006
March 2006
April 2006
May 2006
June 2006
July 2006
August 2006
September 2006
October 2006
November 2006
December 2006
January 2007
February 2007
March 2007
April 2007
May 2007
June 2007
July 2007
August 2007
September 2007
October 2007
November 2007
December 2007
January 2008
February 2008
March 2008
April 2008
May 2008
June 2008
July 2008
August 2008
September 2008
October 2008
November 2008
December 2008
January 2009
February 2009
March 2009
April 2009
May 2009
June 2009
July 2009
August 2009
September 2009
October 2009
November 2009
December 2009
January 2010
February 2010
March 2010
April 2010
May 2010
June 2010
July 2010
August 2010
September 2010
October 2010
November 2010
December 2010
January 2011
February 2011
March 2011
April 2011
May 2011
June 2011
July 2011
August 2011
September 2011
October 2011
November 2011
December 2011
January 2012
February 2012
March 2012
April 2012
May 2012
June 2012
July 2012
August 2012
September 2012
October 2012
December 2012
February 2013
April 2013
June 2013
July 2013
August 2013
March 2014
April 2014
August 2014
October 2015
March 2023