Fred Kaplan takes another shot at being an idiot and in the process earns a much deserved Chowderhead of the Day.
In his latest effort to understand statistics and the Lancet studies he begins:
Initially, I decided to stay out of this controversy. I'd written the first critique of an earlier Lancet/Hopkins study, which estimated that 100,000 Iraqis had died in just the first year of the war. The study's sample was too small, the data-gathering too slipshod, the range of uncertainty so wide as to render the estimate useless.
It's like he lives in a bubble. He was mocked by those who know something about statistical analysis and epidemiological studies such as the Lancet survey in 2004. But he clings to disproved claims that the first study was useless. After all he has got a reputation to maintain as an unqualified hack.
Basically in his mess of an article he cites two supposed flaws with the study. The first, regarding estimates regarding baseline figure of deaths in Iraq per 1000 pre-war used to compare with the rate of deaths since 2003 in order to establish the 'extra' deaths in Iraq demonstrates once more that he has no idea what the hell he is talking. The second supposed flaw comes from a study done by two physicists and an economist, hardly experts in this field. First, the deaths per thousand - Kaplan:
Based on the household surveys, the report estimates that, just before the war, Iraq's mortality rate was 5.5 per 1,000. (That is, for every 1,000 people, 5.5 die each year.) The results also show that, in the three and a half years since the war began, this rate has shot up to 13.3 per 1,000. So, the "excess deaths" amount to 7.8 (13.3 minus 5.5) per 1,000. They extrapolate from this figure to reach their estimate of 655,000 deaths.
However, according to data from the United Nations, based on surveys taken at the time, Iraq's preinvasion mortality rate was 10 per 1,000. The difference between 13.3 and 10.0 is only 3.3, less than half of 7.8.
Well, since the link he supplied to the UN doesn't work, we can not determine how valid the UN number is. However, he says:
(If the Hopkins researchers [the authors of the Lancet study] want to claim that their estimate is more reliable than the United Nations', they will have to prove the point. It is also noteworthy that, if Iraq's preinvasion mortality rate really was 5.5 per 1,000, it was lower than that of almost every country in the Middle East, and many countries in Western Europe.)
Immediately he shows that he has no understanding of the concept of death rates, making the oft-repeated claim without evidence, that is it just doesn't make sense with Western Europe's much higher rates. The death rate has more to do with the age of the population than the health system. Why don't we consult Gilbert Burnham, one the authors of the study, to explain. (The questions come from a Pajamas Media blog):
(Q) The Lancet study uses a baseline mortality rate (the rate during Saddam years) of 5.5/1000 – almost half of the mortality rate of Europe. The mortality rate in the EU is 10.10/1000. Given Europe’s excellent health care, public health infrastructure and, lack of war in the past 60 years, how is it possible that Iraq’s baseline is half that of the EU? Are you simply relying on pre-war publications or was the baseline itself generated by interviews with random clusters?
Burnham: This was a ‘cohort’ study, which means we compared household deaths after the invasion with deaths before the invasion in the same households. The death rates for these comparison households was 5.5/1000/yr.
What we did find for the households as a pre-invasion death rate was essential the same number as we found in 2004, the same number as the CIA gives and the estimate for Iraq by the US Census Bureau.
Death rates are a function of many things—not just health of the population. One of the most important factors in the death rate is the number of elderly in the population. Iraq has few, and a death rate of 5.5/1000/yr in our calculation (5.3 for the CIA), the USA is 8 and Sweden is 11. This is an indication of how important the population structures are in determining death rates. (You might Google ‘population pyramid’ and look at the census bureau site—fascinating stuff.)
Burnham also notes how the numbers they came up with independently coincided with the number used by the U.S. census bureau, further validating their figure
This was a cohort study so we used the same household experiences from 1 Jan 2002- March 2003 as the comparison period. It also came out that this was almost exactly the US Census Bureau estimate as well--which of course made us feel good.
Kaplan apparently is still learning about Googling and such. I guess he could also point to Hungary. I mean they haven't had a war in decades.
(Q from Pajamas Media) The Lancet Study comes up with a post-war mortality rate almost double that Saddam’s Iraq. In fact, it is roughly equivalent to the mortality rate in Hungary is 13/1000. Does that rate seem plausible, given Hungary’s superior infrastructure and almost 50 continuous years of peace? Is it possible that both the pre- and post-war mortality rates are too low? Why not?
Burnham: There are many old people in Hungary , 40% are over age 55 vs. 9.3% in Iraq over 55. That’s the difference.
So Kaplan fails to understand the obvious logic in the 'population pyramid,' and fails to note that the number 5.5 is used by not only by the study - which btw was attained in this study through the use of household surveys - but by the CIA and U.S. census bureau. So the first 'flaw' he cites isn't one. Period.
Second, he goes to his crack team of physicists and an economist to arrive at this conclusion:
... let's look at this study's numbers and why they're almost certainly overstated...
....A joint research team led by physicists Sean Gourley and Neil Johnson of Oxford University and economist Michael Spagat at Royal Holloway University in London noticed the second flaw. In a statement released Thursday (and reported in today's issue of the journal Science), they charged that the Lancet study is "fundamentally flawed"—and in a way that systematically overstates the death toll.
The Lancet study, in its section on methodology, notes that the teams picked the houses they would survey from a "random selection of main streets," defined as "major commercial streets and avenues." (Italics added.) They also chose from a "list of residential streets crossing" those main streets.
The Oxford-Holloway team calls this method "main street bias." They add:
Main street bias inflates casualty rates since conflict events such as car bombs, drive-by shootings, artillery strikes on insurgent positions, and marketplace explosions gravitate toward the same neighborhood types that the [Lancet] researchers surveyed. …
In short, the closer you are to a main road, the more likely you are to die in violent activity. So if researchers only count people living close to a main road, then it comes as no surprise they will over-count the dead.
Whether or not the Hopkins researchers were aware of this flaw, or its importance, is unclear. An exchange of e-mails with Gilbert Burnham, the study's chief researcher, raises some disturbing questions about this matter...
Not so much really, unless the disturbing questions are along the lines of why does Slate allow tripe like your column to be published. As Tim Lambert at Scienceblogs says
I guess that the next time a new physics study comes out Science will ask epidemiologists what they think of it. You see, John Bohannon, the reporter for Science, decided that opinions from a couple of physicists and an economist were more important than getting comments from experts in epidemiology.
Perhaps if Kaplan had spent a little time googling he could have found out the information himself instead of being puzzled why the authors didn't want to respond to this hack. Oh yeah he doesn't know how. (It should also be noted that in several sites that discuss the study it is noted by the authors and/or commentators that the methodology supplied for the public and journalists is a far more simplified 'laymen' version than the one supplied for peer review) Lambert again:
Contrary to Bohannon's they did not avoid small back alleys for safety reasons. And contrary to the claims by SJG, they don't just survey houses close to or on main roads. According to the description above, they don't survey main roads at all, and the houses on cross streets aren't necessarily close to the main road -- a random house was chosen. To the extent that deaths tend to be concentrated on main roads, the survey will produce an under-estimate. Further, while deaths are more likely to occur on a main street, that isn't the relevant characteristic as far as the study is concerned, but where the people lived. If someone dies in because a car bomb is set off in a marketplace, that person will be counted if the Lancet surveys his home. "Main-street bias" exists only to the extent that people living on main streets would be killed in their homes. Deaths from car bombs and drive-by shootings will mainly be from people out in the open, but people in their homes. So while "main-street bias" would produce an under-estimate, it's unlikely to make much difference and it is wrong to call it serious flaw.
And from a commentator at the Pajamas Media site:
...Unfortunately this is not an accurate description of the methodology. Burnham et. al. (the 2006 Lancet article) described describe the household selection thusly:
"The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected."
In other words main streets were EXCLUDED leading to "residential street bias".
All when the streets define a regular grid then EVERY residential street will cut a main street and choosing a main street at random merely fixes the orientation of the residential street (e-w or n-s for example). The houses are then chosen randomly over the length of the residential street...so there will be no 'main street bias'. There is however 'residential street bias' which would presumably lead to an UNDERESTIMATE.
When the streets do not define a regular grid then there will some bias. Exactly how much and if it is sufficient to overcome 'residential street bias' is unclear as it depends on the details of the street layout and the violence differential between main streets and residential streets.
Also, as a commentator notes at the science blogs site, the main street bias - which almost certainly did not occur here - would only produce a sampling bias, not necessarily as Kaplan implies, a mortality bias.
Perhaps people who live on the main streets are more likely to be hit by collateral damage from car bombs. But they spend less time walking on the street, so they are less likely to be hit by stray bullets. etc.
Quantum complexity/financial researcher Neil Johnson has no idea whether "main street sampling bias" implies anything about mortality bias. He has no data to back up his handwaving, and any "simulations" he does will be worth considerably less than the data set he uses for them. And there are no good data sets for this problem...
And Burnham notes one more reason that, if anything, there might be a tendency to underestimate the figure.
...There are always chances that sampling was done in more hot spots, but there is an equal chance that, with a natural human tendency to self-preservation might cause sampling to be the other way to unconsciously sample in cool spots where one might be safer....
Finally, Lambert once more on the supposed main street bias.
if we go by why the Lancet article says, the bias is away from main streets. If we go by what Burnham now says they did there is no main street bias at all. I don't see how any communications between Spagat and co and Burnham and co lets Spagat conclude that they did something other than what they said they did.
No Fred, you have presented nothing at all that would lead someone to conclude that the estimates were " almost certainly overstated." During his thoroughly innumerate evaluation he states the following:
Here lies the danger of studies that overstate a war's death toll. The war's supporters and apologists latch on to the inevitable debunkings and proclaim that really "only 100,000" or "only 200,000" people have died. It's obscene—it sullies and coarsens the political culture—to place the word "only" in front of such numbers.
No one has overstated the death toll. It is his shoddy work that gives ammo to wingnuts and has contributed to the 'sullying and coarsening of the political culture' by allowing them to use his mess as justification in adding 'only' when they talk about the number of killed in Iraq. Kaplan's stubborn, ill-informed, unscientific stance would make a Republican proud.
Why is he so determined to prove these studies, done by people who have academic reputations to protect (and can't simply be spectacularly wrong and ill-formed yet never admit it like Kaplan can)? Obviously Kaplan paid no attention to all the experts who had ridiculed him the first time. Now he steps right into it again and most certainly won't pay attention to the mockery he is about to receive once more. Let's be very clear. It is Kaplan - not the academics and experts who bravely conducted the study - who is enabling the "war's supporters and apologists" by supplying them with baseless 'debunkings.'
There is a host of sites and comments demonstrating how the study has been peer reviewed. From alertnet:
A controversial estimate by public health experts that hundreds of thousands of Iraqis have died because of the March 2003 U.S.-led invasion of Iraq is likely an accurate assessment, researchers said on Saturday.
"Over the last 25 years, this sort of methodology has been used more and more often, especially by relief agencies in times of emergency," said Dr. David Rush, a professor and epidemiologist at Tufts University in Boston.....
"I think this is an extremely credible study," said Michael Intriligator, professor of economics at the University of California at Los Angeles.
The author Burnham once more.
The Lancet had a very detailed peer review, and then we worked closely with an editor to get the paper in its final form. This followed a standard track through the Lancet process--something over which no author has control-
To sum on Kaplan. Neither of his arguments that the study is flawed are credible and he only proves he is completely unqualified to write about the subject - once again.
A commentator at the science blog aptly notes how ridiculous the whole controversy is:
If you are a biased researcher determined to inflate the numbers of dead in Iraq and get published like this, you have to go through a) raising the money, finding the researchers and dealing with the safety problems and then b) create the papers and then c) get them taken on by the most prestigious medical magazine in the UK and then d) run them through a rigorous peer review process in which you cannot at all guarantee that the anonymous critics share your bent world view.
If you are a Republican shill who wants to attack the study, you have to a) go to your computer and b) convince the editorial board of the Wall Street Journal that your position undermines the Left and supports the administration and c) collect your cheque.
Notice any difference here?
Posted by: david tiley
Finally as Professor Burnham says:
I am not fixed on the 654,000 number--it well could be above or below--but statistically we are 95% confident that the true number is in that range. My greater concern is to move away from the number and think of the lives that this represents...
This is an unimaginable disaster and tragedy that was brought about by the criminals in the White House and their supporters. Kaplan, in writing crap like this, does more to shield them by, as he himself says, allowing them to use 'only' before discussing the deaths in Iraq. This too important and too horrible to be in the hands of someone who obviously has precious little understanding or knowledge of the matter. He should just shut up and eat his bowl of chowder.
Now that the Lancet report has been proved to wrong in every conceivable way, I bet you feel pretty stupid now, huh?
Posted by: John | January 10, 2008 at 03:31 PM
The death toll is in the hundreds of thousands regardless, the Lancet survey has definitely not "been proved to [be] wrong in every conceivable way," there is plenty of disagreement and variance in the death toll surveys in Iraq, everything I said in the post was accurate and still is, and you, as they say, are not the brightest candle on the cake.
Posted by: b.h | January 10, 2008 at 03:55 PM