Think a little more before you click "analyze"!
Imagine you're a researcher who has been granted access to a police department's violent crime reports. First, you look for homicides with adult female victims, and you find exactly 50 cases in which a woman was killed by her husband, plus another 50 in which the killer was someone else. (This is a hypothetical example, so I can make up any numbers I want.) Next you look at cases of aggravated assault against women, and come up with thousands, which include only a couple in which the assailant was the victim's husband. What might you infer?
Here's one possible inference: "Assaults on women are rarely fatal, with one major exception. If it's her husband who assaults her, she almost always dies!" After all, there were a total of 52 assaults by husbands in the police records, and 96% of them proved fatal, compared to just 1% (let's say) of the thousands of assaults by other people.
I hope you just thought "Whoa! That's ridiculous!" It would be absurd to infer that assaults by husbands are uniquely likely to be lethal because there's an obvious, highly plausible alternative explanation for the data. A non-fatal act of wife-beating, in the privacy of the home, is unlikely to be investigated and recorded as a criminal offence (1,2), whereas an assault of equal severity by a stranger probably will be. In other words, there's a bias in what gets recorded as an instance of aggravated assault, and the same bias doesn't apply - or applies to a much lesser degree - to homicides.
No experienced researcher would make this sort of inferential mistake, right? Wrong! In the latest issue of the journal Homicide Studies, a group that includes some of the field's leading researchers did precisely that (3). The study in question compared 16,659 incidents of aggravated assault and 876 homicides in which the victims were young children, using an American database called NIBRS (the FBI's National Incident-based Recording System). The aim was to identify factors that affect the lethality of assaults on infants and toddlers, whereas my hypothetical example concerned assaults on women, but the logical flaw is identical.
The results? According to the paper's abstract, "the probability of fatality is significantly influenced by victim and offender characteristics, victim-offender relationship, weapon used, time of incident, and region of the United States in which the incident occurred". Unfortunately, none of this is valid, for the same reason that my imaginary inference about the exceptional lethality of wife assault isn't valid. Their analytic strategy entailed an implicit assumption that the authors apparently never noticed they were making, namely that the criminalized assaults in the NIBRS data are representative of a larger universe of events that meet the criteria of "aggravated assault" but may not have been investigated and recorded as such.
Let me spell out the problem with respect to one exemplary "finding". The authors conclude that an assault on an infant by a birth parent is more likely to be fatal than an assault on an infant by a non-parent. But like wife-beating, parental mistreatment of children happens predominantly behind closed doors and often comes to light only when the victim dies. Moreover, parents are widely granted a discretionary right to discipline their children (even where anti-spanking laws are in effect): a mother who gives her wailing baby a hard smack in a public venue is very unlikely to face criminal charges, whereas a stranger who is annoyed by the wailing would be unwise to assume that he has the same impunity. In short, the chances that an assault on an infant will end up in the NIBRS data are almost certainly lower when the assailant is a parent than otherwise, and if that be so, the conclusion about differential lethality goes out the window. All the paper's other purported results confront similar challenges.
Why would experts fall prey to an error like this? One possibility is that their use of a statistical package to effectuate a multivariate analysis that "controlled" for other potential predictors of lethality, such as "race", weapon, and the age and sex of both perpetrator and victim, obscured the logic of what they were actually doing. The results are presented as regression coefficients and odds ratios, outputs that impart persuasiveness and an air of sophistication. The reviewers apparently missed the fact that the conclusions are nevertheless unfounded, and so did the editor.
In some final remarks about the "limitations" of their study, the authors note that results can be biased by "the dark figure of crime", but in elaborating on this point, they make it clear that they are thinking only of undetected homicides. Undetected homicides are indeed a concern, perhaps especially when the victims are preschoolers (e.g. 4), but it is the biases in what gets recorded as an assault that pose the greater threat to this study's conclusions. And this brings me to a disheartening postscript. I sent my critique - actually a more diplomatically worded version - to the current editor of Homicide Studies (who had just replaced the editor who presided over the paper's review). She replied that she would not publish my letter on the grounds that (a) the original paper had passed expert review, and (b) it was reasonable for the authors to express concern about undetected homicides while ignoring undetected assaults because the journal is Homicide Studies, not Assault Studies! My argument that the paper's inferences are fallacious and its conclusions unwarranted wasn't disputed; it was simply ignored.
If you've read this far, you may think this a fairly nasty ad hominem attack. It's not meant that way. I actually like those of this paper's authors with whom I'm acquainted, and I have thought well of work they have published in the past. Their faulty logic in the present instance shocked me, and the editor's response compounded the shock. But maybe I shouldn't have been so surprised. It's not like this is a unique instance, nor even exceptional. The junior researchers who took the lead on this study are under enormous pressure to publish as much and as quickly as they can, while rewards for taking the time to think critically are less immediate and less certain. Their senior colleagues are under pressure to produce, too.
In 2005, John Ioannidis famously asserted that "most published research findings are false" and laid most of the blame on weak research designs and shoddy statistical inference (5). If you haven't read his paper, please do so. His primary target was medical epidemiology, but the social sciences, which are equally if not more reliant on non-experimental methods than epidemiology, provide plenty of material for a similar critique. Just last week, my wife and collaborator, Gretchen Perry, drew my attention to some papers in which an association between residing with extended family members and child behaviour problems or ineffectual parenting led their authors to infer that a grandmother's presence in the home creates the problems (6,7), a classic example of confusing correlation with causation that would be amusing if it weren't so alarming.
The issue here is much bigger than the fact that unwarranted conclusions muddy our understanding, bigger even than the potential for mischief when those unwarranted conclusions affect policy. The social sciences generate information and understandings that are not only intellectually enriching but of immense practical importance, and yet they are under attack from forces that wish to defund research and replace inconvenient findings with "alternative facts" that suit them better. Central to this agenda is the portrayal of social scientists as biased rather than objective; the last thing we should be doing is providing them with ammunition! We need to be more concerned than ever that our methods are rigorous and our conclusions defensible.
Martin Daly. Hamilton, Ontario. March 25, 2017.
(1) Dobash, R.E. & Dobash, R.P. (1979) Violence against wives. A case against the patriarchy. New York: Free Press.
(2) Palermo, T., Bleck, J. & Peterman, A. (2013) Tip of the iceberg: Reporting and gender-based violence in developing countries. American Journal of Epidemiology 179: 602-612.
(3) Clifford, J.E., Tetzlaff-Bemiller, M.J., Jarvis, J.P., Huff-Corzine, L., Weaver, G.S., & Corzine, J. (2017) Fatal factors for preschoolers: victims, offenders, and context. Homicide Studies 21: 3-20.
(4) Ewigman, B., Kivlahan, C., & Land, G. (1993) The Missouri child fatality study: underreporting of maltreatment fatalities among children younger than five years of age, 1983 through 1986. Pediatrics 91: 330-337.
(5) Ioannidis, J.P.A. (2005) Why most published research findings are false. PLOS Medicine 2(8): e124.
(6) Unger, D.G. & Cooley, M. (1992) Partner and grandmother contact in black and white teen parent families. Journal of Adolescent Health 13: 546-552.
(7) Chase‐Lansdale, P.L., Brooks‐Gunn, J. & Zamsky, E.S. (1994) Young African‐American multigenerational families in poverty: Quality of mothering and grandmothering. Child Development 65: 373-393.
Imagine you're a researcher who has been granted access to a police department's violent crime reports. First, you look for homicides with adult female victims, and you find exactly 50 cases in which a woman was killed by her husband, plus another 50 in which the killer was someone else. (This is a hypothetical example, so I can make up any numbers I want.) Next you look at cases of aggravated assault against women, and come up with thousands, which include only a couple in which the assailant was the victim's husband. What might you infer?
Here's one possible inference: "Assaults on women are rarely fatal, with one major exception. If it's her husband who assaults her, she almost always dies!" After all, there were a total of 52 assaults by husbands in the police records, and 96% of them proved fatal, compared to just 1% (let's say) of the thousands of assaults by other people.
I hope you just thought "Whoa! That's ridiculous!" It would be absurd to infer that assaults by husbands are uniquely likely to be lethal because there's an obvious, highly plausible alternative explanation for the data. A non-fatal act of wife-beating, in the privacy of the home, is unlikely to be investigated and recorded as a criminal offence (1,2), whereas an assault of equal severity by a stranger probably will be. In other words, there's a bias in what gets recorded as an instance of aggravated assault, and the same bias doesn't apply - or applies to a much lesser degree - to homicides.
No experienced researcher would make this sort of inferential mistake, right? Wrong! In the latest issue of the journal Homicide Studies, a group that includes some of the field's leading researchers did precisely that (3). The study in question compared 16,659 incidents of aggravated assault and 876 homicides in which the victims were young children, using an American database called NIBRS (the FBI's National Incident-based Recording System). The aim was to identify factors that affect the lethality of assaults on infants and toddlers, whereas my hypothetical example concerned assaults on women, but the logical flaw is identical.
The results? According to the paper's abstract, "the probability of fatality is significantly influenced by victim and offender characteristics, victim-offender relationship, weapon used, time of incident, and region of the United States in which the incident occurred". Unfortunately, none of this is valid, for the same reason that my imaginary inference about the exceptional lethality of wife assault isn't valid. Their analytic strategy entailed an implicit assumption that the authors apparently never noticed they were making, namely that the criminalized assaults in the NIBRS data are representative of a larger universe of events that meet the criteria of "aggravated assault" but may not have been investigated and recorded as such.
Let me spell out the problem with respect to one exemplary "finding". The authors conclude that an assault on an infant by a birth parent is more likely to be fatal than an assault on an infant by a non-parent. But like wife-beating, parental mistreatment of children happens predominantly behind closed doors and often comes to light only when the victim dies. Moreover, parents are widely granted a discretionary right to discipline their children (even where anti-spanking laws are in effect): a mother who gives her wailing baby a hard smack in a public venue is very unlikely to face criminal charges, whereas a stranger who is annoyed by the wailing would be unwise to assume that he has the same impunity. In short, the chances that an assault on an infant will end up in the NIBRS data are almost certainly lower when the assailant is a parent than otherwise, and if that be so, the conclusion about differential lethality goes out the window. All the paper's other purported results confront similar challenges.
Why would experts fall prey to an error like this? One possibility is that their use of a statistical package to effectuate a multivariate analysis that "controlled" for other potential predictors of lethality, such as "race", weapon, and the age and sex of both perpetrator and victim, obscured the logic of what they were actually doing. The results are presented as regression coefficients and odds ratios, outputs that impart persuasiveness and an air of sophistication. The reviewers apparently missed the fact that the conclusions are nevertheless unfounded, and so did the editor.
In some final remarks about the "limitations" of their study, the authors note that results can be biased by "the dark figure of crime", but in elaborating on this point, they make it clear that they are thinking only of undetected homicides. Undetected homicides are indeed a concern, perhaps especially when the victims are preschoolers (e.g. 4), but it is the biases in what gets recorded as an assault that pose the greater threat to this study's conclusions. And this brings me to a disheartening postscript. I sent my critique - actually a more diplomatically worded version - to the current editor of Homicide Studies (who had just replaced the editor who presided over the paper's review). She replied that she would not publish my letter on the grounds that (a) the original paper had passed expert review, and (b) it was reasonable for the authors to express concern about undetected homicides while ignoring undetected assaults because the journal is Homicide Studies, not Assault Studies! My argument that the paper's inferences are fallacious and its conclusions unwarranted wasn't disputed; it was simply ignored.
If you've read this far, you may think this a fairly nasty ad hominem attack. It's not meant that way. I actually like those of this paper's authors with whom I'm acquainted, and I have thought well of work they have published in the past. Their faulty logic in the present instance shocked me, and the editor's response compounded the shock. But maybe I shouldn't have been so surprised. It's not like this is a unique instance, nor even exceptional. The junior researchers who took the lead on this study are under enormous pressure to publish as much and as quickly as they can, while rewards for taking the time to think critically are less immediate and less certain. Their senior colleagues are under pressure to produce, too.
In 2005, John Ioannidis famously asserted that "most published research findings are false" and laid most of the blame on weak research designs and shoddy statistical inference (5). If you haven't read his paper, please do so. His primary target was medical epidemiology, but the social sciences, which are equally if not more reliant on non-experimental methods than epidemiology, provide plenty of material for a similar critique. Just last week, my wife and collaborator, Gretchen Perry, drew my attention to some papers in which an association between residing with extended family members and child behaviour problems or ineffectual parenting led their authors to infer that a grandmother's presence in the home creates the problems (6,7), a classic example of confusing correlation with causation that would be amusing if it weren't so alarming.
The issue here is much bigger than the fact that unwarranted conclusions muddy our understanding, bigger even than the potential for mischief when those unwarranted conclusions affect policy. The social sciences generate information and understandings that are not only intellectually enriching but of immense practical importance, and yet they are under attack from forces that wish to defund research and replace inconvenient findings with "alternative facts" that suit them better. Central to this agenda is the portrayal of social scientists as biased rather than objective; the last thing we should be doing is providing them with ammunition! We need to be more concerned than ever that our methods are rigorous and our conclusions defensible.
Martin Daly. Hamilton, Ontario. March 25, 2017.
(1) Dobash, R.E. & Dobash, R.P. (1979) Violence against wives. A case against the patriarchy. New York: Free Press.
(2) Palermo, T., Bleck, J. & Peterman, A. (2013) Tip of the iceberg: Reporting and gender-based violence in developing countries. American Journal of Epidemiology 179: 602-612.
(3) Clifford, J.E., Tetzlaff-Bemiller, M.J., Jarvis, J.P., Huff-Corzine, L., Weaver, G.S., & Corzine, J. (2017) Fatal factors for preschoolers: victims, offenders, and context. Homicide Studies 21: 3-20.
(4) Ewigman, B., Kivlahan, C., & Land, G. (1993) The Missouri child fatality study: underreporting of maltreatment fatalities among children younger than five years of age, 1983 through 1986. Pediatrics 91: 330-337.
(5) Ioannidis, J.P.A. (2005) Why most published research findings are false. PLOS Medicine 2(8): e124.
(6) Unger, D.G. & Cooley, M. (1992) Partner and grandmother contact in black and white teen parent families. Journal of Adolescent Health 13: 546-552.
(7) Chase‐Lansdale, P.L., Brooks‐Gunn, J. & Zamsky, E.S. (1994) Young African‐American multigenerational families in poverty: Quality of mothering and grandmothering. Child Development 65: 373-393.