Angela L. Duckworth's IQ-Motivation Study : Another Bad Apple

The paper "Role of test motivation in intelligence testing" (2011) by Duckworth et al. is widely cited as an anti-IQ evidence. Some anti-IQ supporters wrongly believe that the study invalidates the "IQ thesis". In fact, there is a huge misunderstanding. As Duckworth et al. pointed out :

It is important not to overstate our conclusions. For all measured outcomes in Study 2, the predictive validity of intelligence remained statistically significant when controlling for the nonintellective traits underlying test motivation. Moreover, the predictive validity of intelligence was significantly stronger than was the predictive validity of test motivation for academic achievement. In addition, both Studies 1 and 2 indicate that test motivation is higher and less variable among participants who are above-average in measured IQ. These findings imply that earning a high IQ score requires high intelligence in addition to high motivation. Lower IQ scores, however, might result from either lower intelligence or lack of motivation. Thus, given closer-to-maximal performance, test motivation poses a less serious threat to the internal validity of studies using higher-IQ samples, such as college undergraduates, a popular convenience sample for social science research (43). Test motivation as a third-variable confound is also less likely when experimenters provide substantial performance-contingent incentives or when test results directly affect test takers (e.g., intelligence tests used for employment or admissions decisions).

But what is the relation with "g" (i.e. the general intelligence factor) ? There is no reason to think the IQ gains last after the test-takers receive their pay (i.e. the incentives). In this case, the study is worth nothing. It tells us nothing about the malleability of IQ by means of social programs, which all failed to produce lasting effects on IQ. And given that the two groups of subjects have been subjected to different conditions, external bias may have occurred (Jensen, 1980, ch. 12). Thus, conceptually, if IQ score can be achieved through different pathways, as the authors imply (e.g., either intelligence or motivation), then measurement bias should be detectable. But empirically, this is not what is usually seen. Now we come to the technical problem with Duckworth's study. The blogger Statsquatch gives the first blow to Duckworth's IQ-motivation study :

The paper's meta-analysis of the 46 studies was very good but they neglected to make a "forest plot" that graphs the effect sizes and confidence intervals from each study.  So I did it.  The size of the effect circle is proportional to the inverse of the effect's standard error, i.e., bigger studies that get more weight have bigger circles.

Two things are apparent: the study outcomes are highly variable, i.e., they are heterogeneous, and there are only three large experiment (2, 3, and 4 in the graph) that showed motivation leading to a large improvement in IQ score and hence are very influential.

The three experiments were run on special ed kids and written up in one paper by Bruening and Zella (1978) while the first author was at the Oakdale Center for Developmental Disabilities.  A few years later Bruening admitted to fabricating data for other experiments on retarded kids at the same center and the case became a text book example of scientific fraud.  Although there were no allegations of fraud on the 1978 paper, I re-ran the meta-analysis with out Bruening's data and found the the estimate (using the random effects model) was now 0.48 SD and was no longer significant ( p = .07).  This makes me uneasy about accepting Duckworth's results without further replication.

Sackett et al. "High-Stakes Testing in Higher Education and Employment" (2008) give the second blow. Here's what they had to say about Duckworth's previous study "Self-Discipline Outdoes IQ in Predicting Academic Performance of Adolescents" (2005) on motivation/self-discipline :

“They applied a range restriction correction to one of the six outcome measures (GPA) and reported that whereas the corrected IQ–GPA correlation (.49) was larger than the uncorrected value (.32), it remained lower than the self-discipline–GPA correlation (.67). Although this is true, note that their conclusion (self-discipline accounts for more than twice as much variance as IQ) no longer holds after one takes range restriction into account. In addition, we applied range restriction corrections to other outcomes; in the case of predicting procrastination (as measured by the time homework was begun), for example, IQ had a higher correlation (-.28 corrected, -.18 observed) with procrastination than did self-discipline (-.26) after we corrected for range restriction, which is clearly at odds with the authors’ conclusion.”

There is, however, something that nobody seems to notice. The biggest problem with Duckworth's motivation theory is to imply that motivation is a malleable trait (i.e. modifiable by means of parental education). That is to say, the study on the role of motivation/self-discipline in school achievement is relevant only on the assumptions that motivation is 1) largely malleable and 2) largely independent of IQ. For the first proposition, most behavioral traits are under genetic influences (Rowe, 1994). For the second proposition, Jensen (1980, p. 322) for example, mentioned that it is success (failure) itself that enhances (dampens) motivation.

And this is the final blow to Duckworth's motivation theory. If motivation is highly malleable and more important than IQ in predicting school achievement, we must expect that parental SES (socio-economic status) is more important than IQ in predicting, for instance, the probability of getting a college degree because high SES parents may foster motivation of their children by providing better cognitive stimulation. However, when we compare the independent role of IQ and the independent role of SES on school achievement, we can notice that IQ (when parental SES is held constant) is much more important than parental SES (when IQ is held constant) in predicting the probability of getting a college degree and the probability of permanently dropping out of high school.

Of the whites who dropped out never to return, only three-tenths of 1 percent met a realistic definition of the gifted-but-disadvantaged dropout (top quartile of IQ, bottom quartile of socioeconomic background.) (The Bell Curve, p. 148)

In terms of this figure, a student with very well-placed parents, in the top 2 percent of the socioeconomic scale, had only a 40 percent chance of getting a college degree if he had only average intelligence. A student with parents of only average SES – lower middle class, probably without college degrees themselves – who is himself in the top 2 percent of IQ had more than a 75 percent chance of getting a degree. (The Bell Curve, p. 153)

The effect of parental SES is not sufficient enough to support the motivation theory. We must recall that IQ is an important predictor of longevity, car accident, work accident (Gottfredson & Deary, 2004). IQ also predicts - beside other factors - lower crime rate, illegitimate birth rate, divorce rate, and so on (Gottfredson, 1997, 2002). Motivation can't account for all of these social outcomes.