Beyond the Evidence Given:
A Critical Commentary on Critical Links
Ellen Winner
Department of Psychology, Boston College
Project Zero, Harvard Graduate School of Education
Lois Hetland
Project Zero, Harvard Graduate School of Education
November, 2002
To appear in Arts Education Policy Review
In what sense are the links summarized in Critical Links: Learning in the Arts and Student Academic and Social Development, actually "critical"? Are they critical in the scientific sense of that word, meaning "characterized by careful and exact evaluation and judgment?" Or "critical" meaning "essential, but in short supply"? (1) In what follows, we compare some of the sweeping claims made in some of this volume's essays with the more nuanced summaries of the actual studies on which these claims are based. The claims go beyond the evidence summarized in this volume. We urge the reader to examine the results of each study summary carefully, along with the commentary provided. In these two sections readers can usually learn about the limitations of each study. We intend no criticism of the studies by pointing out that they have limitations--all studies, even the most rigorous, are limited. Our concern is that interpretive claims have been made about the studies that ignore important limitations. As a result, casual readers may come to believe that a small dose of the arts is all that is needed to improve students' thinking skills, social skills, school retention, and academic self-concept. Such a conclusion is simply not scientifically based.
The problems begin in the introductory essay authored by Richard Deasy (pp. iii-iv). We read that the essayists "agree that the Compendium studies suggest that well-crafted arts experiences produce positive academic and social effects (p. iii). The reader should note that a causal claim is being made here--the claim that studying the arts causes academic and social skills to improve. But a careful reading shows that many of the studies neither claim nor support a causal relationship, as discussed below.
In the bookend to Deasy's opening, the volume's concluding essay, entitled "The Arts and the Transfer of Learning," James Catterall states that the studies chosen for this compendium met strict criteria for "their ability to make causal suggestions" (p. 154). Respectfully, we must disagree. Both of us (EW and LH) participated in the creation of this volume by selecting and summarizing the dance, visual arts, music, and multi-arts studies. We included all studies that we believed were well-enough designed to shed light on the relationship between arts learning, on the one hand, and academic and/or social learning, on the other. But some of the studies selected were well-designed and purely correlational. A correlation between some form of study (here, the arts) and some kind of outcome (here, social or cognitive) offers no information about causality. Of course, two factors that are correlated may be related causally, though we cannot know whether or how: A may cause B (arts study may enhance skills); B may cause A (that is, students with strong skills may choose to study the arts). But two correlated factors may also be causally unrelated since a third factor may cause both. For example, parents may push their children to work hard in academic classes and to study the arts; or schools with good arts programs may also have good academic programs, leading to students excelling in both.
The closing essay goes on to state that the studies "all" show evidence of transfer (p. 154). Let's take a close look at the claims laid out in his Figure 1 (pp. 152-153), and then look back at the studies summarized. Causal connections have indeed been demonstrated in some areas: music enhances certain kinds of spatial reasoning but not others; and classroom drama enhances an array of verbal abilities. Our concerns are with the claims made for the transfer effects of exposure to visual arts, dance, and multi-arts programs.
Visual Arts
Catterall lists four links between visual arts and verbal skills. A look at the four studies summarized shows that indeed, reasoning about art transfers to reasoning about scientific images; and instruction in visual art transfers to reading readiness (although not to actual reading). But the claim that studying drawing leads to improvement in the content and organization of writing does not follow from the DeJarnette study on which Catterall bases that claim. The reader who turns to page 141 will see that DeJarnette showed that drawing can be a useful means of assessing student learning of history when it is used in conjunction with writing: when students were allowed to both write and draw, they revealed more knowledge of history than when they wrote but did not draw. It is misleading to suggest that this study showed that learning to draw improves the content and organization of writing.
Finally, the link between visualization training and "sophisticated reading skills" boils down to the finding that two nine-year-old reluctant readers became better readers over the course of nine weeks in which they read stories, drew their visual impressions of the stories, and talked about the relationship between their pictures and the stories. Because there was no comparison group of students spending as much time reading and talking about stories but not making images, there is no way to determine whether making the images was causally related to the reading improvement found. As one of us (EW) wrote in our commentary on this study, "this is a hypothesis-generating study." The study did not test the hypothesis that drawing helps reluctant readers become better readers; only an experimental design could ascertain whether this hypothesis holds up.
Terry Baker, in his reflections on the visual arts studies in this volume (pp. 145-150), also raises an important caution about this study. Just because students make graphic images does not allow the conclusion that they are studying the arts.
Dance
Similar concerns arise about the eleven "links" Catterall identifies between dance and social or academic outcomes. Two studies found that dance instruction resulted in greater creativity as measured by the Torrance Test of Creative Thinking (Minton and Kim). One study found that moving helped beginning readers learn sound-symbol relationships. (This finding raises an interesting question about cause: Could it be that "enacting through dance" supports the development of verbal skills in ways similar to those caused by "enacting through drama," one of the findings of Podlozny's [2000] meta-analysis of classroom drama?) And, along with rich descriptions from their qualitative analysis, three studies generated hypotheses for future research but, responsibly, made no claims about causal relationships. Thus, of the list of eleven "links," only two have any demonstration of cause: reading readiness (one study) and creativity (two studies). Readers must judge for themselves how "reliably" a result is demonstrated by one or two studies. But for the other seven links suggested in Figure 1, the researchers did not claim cause. Rather, the study authors suggested reasonable areas to seek demonstrated effects in future studies.
Multi-Arts Programs
The number of causal outcomes listed in Figure 1 from studying multiple forms of art is staggering. We find six cognitive outcomes said to result from programs that integrate the arts with academics (reading, verbal skills, math skills, creative thinking, achievement motivation, cognitive engagement), 13 cognitive or motivational outcomes said to result from intensive arts study (self-confidence, risk-taking, paying attention, persevering, empathy, self-initiating, persistence, ownership of learning, collaboration, leadership, reduced dropout, educational aspirations, and higher order thinking skills), and four cognitive outcomes from arts-rich school environments (creativity, engagement, a range of personal and social developments, and again, higher order thinking skills).
This sounds almost too good to be true. We can only advise readers once again to look closely at the study summaries. Most contain cautionary notes about how much we can really conclude. Here are just a few examples.
With respect to the study by Burton, Horowitz, and Abeles (p. 66), EW writes that the study showed that "students in arts-rich schools scored higher in creativity and… academic self-concept than students in schools without that level of the arts. Arts-rich schools also had more innovative teachers… This study is correlational in design and does not allow causal conclusions." The commentary goes on to suggest the possibility that students in arts-rich schools scored higher because they also had better teachers.
With respect to the study by Catterall that is summarized on pp. 68-69, we read that the study shows that high arts involved students do better on a range of academic measures. But "because this study is correlational in design, it does not allow us to conclude that arts involvement caused academic achievement to rise. It is equally possible that high academic students choose to involve themselves in the arts" (p. 68). The casual reader might miss this and only notice the large font quotation stating that "High arts students…earned better grades and scores." This "link" can only be interpreted when readers know that the study was correlational and not experimental in design and thus is silent with respect to causality. The very same point is made (in small font) about the study by Catterall, Chapleau, and Iwanaga that is summarized on pp. 70-71 and the study by Catterall and Waldorf that is summarized on pp. 72-3. The large font quotations state the relationships found without cautioning the reader against concluding that there is a causal link. The decontextualized excerpts throughout the volume create a "quick read" text that tells a consistently inflated story about arts effects on academic and social outcomes.
Does studying the arts reduce high school dropout rates? Yes, according to Catterall's Figure 1. But take a close look at the study on this topic by Barry, Taylor, and Walls (1990) summarized on pp. 74-5. This study reported that out of 40 students classified as at risk for high school dropout and asked whether an arts course had affected their decision to remain in school, most said yes. While this is suggestive, can we conclude with any certainty that what the students said is more than their perception? To demonstrate that the arts reduce drop out, we would need to compare actual dropout rates in schools with and without arts programs, matched in all other important variables. Or we could compare students who drop out with those who don't to see if dropouts differ uniquely in their lack of arts involvement. A study by Mahoney and Cairns (1997) summarized on pp. 80-81 did just this and found that high school dropouts were more likely than those who stayed in school to have had no involvement in extracurricular arts. But these students also had low involvement in extracurricular athletics and in vocational training. Thus, the most parsimonious conclusion is that students who drop out are students who are disengaged from school prior to dropping out.
Does studying multiple forms of arts improve academic performance? Look at the study by Harland et al. (2000) summarized on pp. 76-77. This study was carried out in the UK and found that when social class and prior attainment in the arts were controlled, there was no evidence that the arts improve performance on academic national exams. Yet the large font quotation once again inflates the case, stating that "students performing well in at least one art form reported a wide range of positive effects from arts education (p. 77). Indeed, students did report that they learned much from the arts, including "thinking skills." But the actual test of this question--how students performed on their national exams--failed to demonstrate that those who had studied more arts did better on their exams.
Finally, we note the study by Winner and Cooper (2002), summarized on pp. 92-93. Three meta-analyses of studies examining the link between arts and academics were reported. The results summarized state that the "correlational" meta-analyses (meaning the meta-analyses performed on correlational studies) were significant. Yes, this is true. Students in the US who study the arts are the same students who do well in school and on standardized tests. But the meta-analyses of the experimental studies (with pre- and post-tests, and control groups) did not demonstrate in a statistically reliable way that studying the arts causes verbal or mathematical skills to improve. Only experimental designs can test causal hypotheses.
Catterall concludes that we need more research on this topic "if we want to understand transfer in its full cognitive glory" (p. 157). By putting it in this way, he assumes the results that such studies would show. Further research is surely necessary, but that research is necessary to understand just where transfer occur, if it does, and where it does not. We should not presume that more research will confirm the existence of transfer.
We agree wholeheartedly with one of the final points made in the closing essay--that we need to look more closely at learning in the arts. Catterall believes this will help us better understand transfer. If we can demonstrate that students who learn more in an art form achieve greater transfer to another domain as compared to students who have learned less in the art form, this would indeed provide support for a causal link between learning in the arts and learning in the non-arts domain.
But we also believe there is another reason to examine learning in the arts. It is important to understand just what kinds of cognitive, affective, and social skills are taught and learned in arts classes. Are these the same kinds of skills learned in academic courses? Are some unique to the arts? If we can identify the cognitive and social skills taught in arts classes, especially those taught uniquely or especially well, we will have built a strong argument for the importance of arts education that does not treat the arts as handmaidens to reading, writing, and arithmetic.
We are currently carrying out such a study. Supported by the J. Paul Getty Trust, we are analyzing the kinds of learning and understanding that are fostered in high school visual arts classes. What we are finding may surprise non-artists. Students, of course, are taught technical skills. In drawing class they are taught perspective; in painting class they are taught color mixing; in ceramics class they are taught to use the wheel. But we find that while technical skills are taught, students are simultaneously being taught a host of other skills. Among other things, they are being helped to reflect about their work; to evaluate their work and that made by others; to learn from their mistakes; and to see in new ways. These are important skills to learn. Perhaps they transfer; perhaps they do not. Most likely some can transfer when particular instructional practices are employed. This is for future research to examine. And we hope to carry out such research. But if we value the kinds of skills that we have identified as taught in visual arts classes, then arts advocates will have an argument for the importance of arts education that is based on the core of learning in the arts.
American Heritage Dictionary: Second College Edition (Boston: Houghton Mifflin, 1985).
|
