Intuition and Domain Knowledge

Can you guess what the graphs below show? I’ll give you a couple of hints: (1) each graph measures performance on a different task, (2) one pair of bars in each graph—left or right—represents participants who used their intuition on the task, while the other pair of bars represents folks who used an analytical approach, and (3) one shading represents participants with low domain knowledge while the other represents participants with high domain knowledge (related to the actual task).

It will actually help you to take a moment and go ahead and guess how you would assign those labels, given the little information I have provided. Is the left pair of bars in each graph the “intuitive approach” or the “analytical approach”? Are the darker shaded bars in each graph “high knowledge” participants or “low knowledge” participants?

When Can I Trust My Gut?

A 2012 study by Dane, et. al, published in the journal Organizational Behavior and Human Decision Processes, sets out to address the “scarcity of empirical research spotlighting the circumstances in which intuitive decision making is effective relative to analytical decision making.”

To do this, the researchers conducted two experiments, both employing “non-decomposable” tasks—i.e., tasks that required intuitive decision making. The first task was to rate the difficulty (from 1 to 10) of each of a series of recorded basketball shots. The second task involved deciding whether each of a series of designer handbags was fake or authentic.

Why these tasks? A few snippets from the article can help to answer that question:

Following Dane and Pratt (2007, p. 40), we view intuitions as “affectively-charged judgments that arise through rapid, nonconscious, and holistic associations.” That is, the process of intuition, like nonconscious processing more generally, proceeds rapidly, holistically, and associatively (Betsch, 2008; Betsch & Glöckner, 2010; Sinclair, 2010). [Footnote: “This conceptualization of intuition does not imply that the process giving rise to intuition is without structure or method. Indeed, as with analytical thinking, intuitive thinking may operate based on certain rules and principles (see Kruglanski & Gigerenzer, 2011 for further discussion). In the case of intuition, these rules operate largely automatically and outside conscious awareness.”]

As scholars have posited, analytical decision making involves basing decisions on a process in which individuals consciously attend to and manipulate symbolically encoded rules systematically and sequentially (Alter, Oppenheimer, Epley, & Eyre, 2007).

We viewed [the basketball] task as relatively non-decomposable because, to our knowledge, there is no universally accepted decision rule or procedure available to systematically break down and objectively weight the various elements of what makes a given shot difficult or easy.

We viewed [the handbag] task as relatively non-decomposable for two reasons. First, although there are certain features or clues participants could attend to (e.g., the stitching or the style of the handbags), there is not necessarily a single, definitive procedure available to approach this task . . . Second, because participants were not allowed to touch any of the handbags, they could not physically search for what they might believe to be give-away features of a real or fake handbag (e.g., certain tags or patterns inside the handbag).

Results

Canvas not supported.

As you can see in the graphs at the right (hover for expertise labels), there was a fairly significant difference in both tasks between low- and high-knowledge participants when those participants approached the task using their intuition. In contrast, high- and low-knowledge subjects in the analysis condition in each experiment did not show a significant difference in performance. (The decline in performance of the high-knowledge participants from the Intuition to the Analysis conditions was only significant in the handbag experiment.)

It is important to note that subjects in the analysis conditions (i.e., those who approached each task systematically) were not told what factors to look for in carrying out their analyses. For the basketball task, the researchers simply “instructed these participants to develop a list of factors that would determine the difficulty of a basketball shot and told them to base their decisions on the factors they listed.” For the handbag task, “participants in the analysis condition were given 2 min to list the features they would look for to determine whether a given handbag is real or fake and were told to base their decisions on these factors.”

Also consistent across both experiments was the fact that low-knowledge subjects performed better when approaching the tasks systematically than when using their intuition. For high-knowledge subjects, the results were the opposite. They performed better using their intuition than using a systematic analysis (even though the ‘system’ part of ‘systematic’ here was their own system!).

In addition, while the combined effects of approach and domain knowledge were significant, the approach (intuition or analysis) by itself did not have a significant effect on performance one way or the other in either experiment. Domain knowledge, on the other hand, did have a significant effect by itself in the basketball experiment.

Any Takeaways for K–12?

The clearest takeaway for me is that while knowledge and process are both important, knowledge is more important. Even though each of the tasks was more “intuitive” (non-decomposable) than analytical in nature, and even when the approach taken to the task was “intuitive,” knowledge trumped process. Process had no significant effect by itself. Knowing stuff is good.

Second, the results of this study are very much in line with what is called the ‘expertise reversal effect’:

Low-knowledge learners lack schema-based knowledge in the target domain and so this guidance comes from instructional supports, which help reduce the cognitive load associated with novel tasks. If the instruction fails to provide guidance, low-knowledge learners often resort to inefficient problem-solving strategies that overwhelm working memory and increase cognitive load. Thus, low-knowledge learners benefit more from well-guided instruction than from reduced guidance.

In contrast, higher-knowledge learners enter the situation with schema-based knowledge, which provides internal guidance. If additional instructional guidance is provided it can result in the processing of redundant information and increased cognitive load.

Finally, one wonders just who it is we are thinking about more when we complain, especially in math education, that overly systematized knowledge is ruining the creativity and motivation of our students. Are we primarily hearing the complaints of the 20%—who barely even need school—or those of the children who really need the knowledge we have, who need us to teach them?




ResearchBlogging.org Dane, E., Rockmann, K., & Pratt, M. (2012). When should I trust my gut? Linking domain expertise to intuitive decision-making effectiveness Organizational Behavior and Human Decision Processes, 119 (2), 187-194 DOI: 10.1016/j.obhdp.2012.07.009

Telling vs. No Telling

So, with that in mind, let’s move on to just one of the dichotomies in education, that of “telling” vs. “no telling,” and I hope the reader will forgive my leaving Clarke’s paper behind. I recommend it to you for its international perspective on what we discuss below.

“Reports of My Death Have Been Greatly Exaggerated”

We should start with something that educators know but people outside of education may not: there can be a bit of an incongruity, shall we say, between what teachers want to happen in their classrooms, what they say happens in their classrooms, and what actually happens there. Given how we talk and what we talk about on social media—and even in face-to-face conversations—and the sensationalist tendencies of media reports about education, an outsider could be forgiven, I think, for assuming that teachers have been moving en masse away from the practice of explicit instruction.

There is a large body of research which would suggest that this assumption is almost certainly “greatly exaggerated.”

Typical of this research is a small 2004 study (PDF download) in the U.K. which found that primary classrooms in England remained places full of teacher talk and “low-level” responding by students, despite intentions outlined in the 1998–1999 National Literacy and National Numeracy Strategies. The graph at the right, from the study, shows the categories of discourse observed and a sense of their relative frequencies.

John Goodlad made a similar and more impactful observation in his much larger study of over 1,000 classrooms across the U.S. in the mid-80s (I take this quotation from the 2013 edition of John Hattie’s book Visible Learning, where more of the aforementioned research is cited):

In effect, then, the modal classroom configurations which we observed looked like this: the teacher explaining or lecturing to the total class or a single student, occasionally asking questions requiring factual answers; . . . students listening or appearing to listen to the teacher and occasionally responding to the teacher’s questions; students working individually at their desks on reading or writing assignments.

Thus, despite what more conspiracy-oriented opponents of “no telling” sometimes suggest, the monotonic din of “understanding” and “guide on the side” and “collaboration” we hear today—and have heard for decades—is not the sound of a worldview that has, in practice, taken over education. Rather, it is one of a seemingly quixotic struggle on the part of educators to nudge each other—to open up more space for students to exercise independent and critical thinking. This a finite space, and something has to give way.

Research Overwhelmingly Supports Explicit Instruction

Teacher as
Activator
dTeacher as Facilitatord
Teaching students self-verbalization0.76Inductive Teaching0.33
Teacher clarity0.75Simulation and gaming0.32
Reciprocal teaching0.74Inquiry-based teaching0.31
Feedback0.74Smaller classes0.21
Metacognitive strategies0.67Individualised instruction0.22
Direct instruction0.59Web-based learning0.18
Mastery learning0.57Problem-based learning0.15
Providing worked examples0.57Discovery method (math)0.11

On the other hand, it is manifestly clear from the research literature that, when student achievement is the goal, explicit instruction has generally outperformed its less explicit counterpart.

The table at the left, taken from Hattie’s book referenced above, directly compares the effect sizes of various explicit and indirect instructional protocols, gathered and interpreted across a number of different meta-analyses in the literature.

Results like these are not limited to the K–12 space, nor do they involve only the teaching of lower-level skills or teaching in only in well-structured domains, such as mathematics. These are robust results across many studies and over long periods of time.

And while research supporting less explicit instructional techniques is out there (as obviously Hattie’s results also attest), there is much less of it—and certainly far less than one would expect given the sheer volume of rhetoric in support of such strategies. On this point, it is worth quoting Sigmund Tobias at some length, from his summarizing chapter in the 2009 book Constructivist Instruction: Success or Failure?:

When the AERA 2007 debate was organized, I described myself as an eclectic with respect to whether constructivist instruction was a success or failure, a position I also took in print earlier (Tobias, 1992). The constructivist approach of immersing students in real problems and having them figure out solutions was intuitively appealing. It seemed reasonable that students would feel more motivated to engage in such activities than in those occurring in traditional classrooms. It was, therefore, disappointing to find so little research documenting increased motivation for constructivist activities.

A personal note may be useful here. My Ph.D. was in clinical psychology at the time when projective diagnostic techniques in general, and the Rorschach in particular, were receiving a good deal of criticism. The logic for these techniques was compelling and it seemed reasonable that people’s personality would have a major impact on their interpretation of ambiguous stimuli. Unfortunately, the empirical evidence in support of the validity of projective techniques was largely negative. They are now a minor element in the training of clinical psychologists, except for a few hamlets here or there that still specialize in teaching about projective techniques.

The example of projective techniques seems similar to the issues raised about constructivist instruction. A careful reading and re-reading of all the chapters in this book, and the related literature, has indicated to me that there is stimulating rhetoric for the constructivist position, but relatively little research supporting it. For example, it is encouraging to see that Schwartz et al. (this volume) are conducting research on their hypothesis that constructivist instruction is better for preparing individuals for future learning. Unfortunately, as they acknowledge, there is too little research documenting that hypothesis. As suggested above, such research requires more complex procedures and is more time consuming, for both the researcher and the participants, than procedures advocated by supporters of explicit instruction. However, without supporting research these remain merely a set of interesting hypotheses.

In comparison to constructivists, advocates for explicit instruction seem to justify their recommendations more by references to research than rhetoric. Constructivist approaches have been advocated vigorously for almost two decades now, and it is surprising to find how little research they have stimulated during that time. If constructivist instruction were evaluated by the same criterion that Hilgard (1964) applied to Gestalt psychology, the paucity of research stimulated by that paradigm should be a cause for concern for supporters of constructivist views.

Both the Problem and the Solution

So, it seems that while a “telling” orientation is better supported by research, it is also identified as a barrier, if not the barrier, to progress. And it seems that a lot of our day-to-day struggle with the issue centers around the negative consequences of continued unsuccessful attempts at resolving this paradox.

Yet perhaps we should see that this is not a paradox at all. Of course it is a problem when students learn to rely heavily on explicit instruction to make up their thinking, and it is perfectly appropriate to find ways of punching holes in teacher talk time to reduce the possibility of this dependency. But we could also research ways of tackling this explicitly—differentiating ways in which explicit instruction can solicit student inquiry or creativity and ways in which it promotes rule following, for example.

It is at least worth considering that some of our problems—particularly in mathematics education—have less to do with explicit instruction and more to do with bad explicit instruction. If dealing with instructional problems head on is more effective (even those that are “high level,” such as creativity and critical thinking), then we should be making the sacrifices necessary to give teachers the resources and training required to meet those challenges, explicitly.

Interleaving

research

Inductive teaching or learning, although it has a special name, happens all the time without our having to pay any attention to technique. It is basically learning through examples. As the authors of the paper we’re discussing here indicate, through inductive learning:

Children . . . learn concepts such as ‘boat’ or ‘fruit’ by being exposed to exemplars of those categories and inducing the commonalities that define the concepts. . . . Such inductive learning is critical in making sense of events, objects, and actions—and, more generally, in structuring and understanding our world.

The paper describes three experiments conducted to further test the benefit of interleaving on inductive learning (“further” because an interleaving effect has been demonstrated in previous studies). Interleaving is one of a handful of powerful learning and practicing strategies mentioned throughout the book Make It Stick: The Science of Successful Learning. In the book, the power of interleaving is highlighted by the following summary of another experiment involving determining volumes:

Two groups of college students were taught how to find the volumes of four obscure geometric solids (wedge, spheroid, spherical cone, and half cone). One group then worked a set of practice problems that were clustered by problem type . . . The other group worked the same practice problems, but the sequence was mixed (interleaved) rather than clustered by type of problem . . . During practice, the students who worked the problems in clusters (that is, massed) averaged 89 percent correct, compared to only 60 percent for those who worked the problems in a mixed sequence. But in the final test a week later, the students who had practiced solving problems clustered by type averaged only 20 percent correct, while the students whose practice was interleaved averaged 63 percent.

The research we look at in this post does not produce such stupendous results, but it is nevertheless an interesting validation of the interleaving effect. Although there are three experiments described, I’ll summarize just the first one.

Discriminative-Contrast Hypothesis

But first, you can try out an experiment like the one reported in the paper. Click start to study pictures of different bird species below. There are 32 pictures, and each one is shown for 4 seconds. After this study period, you will be asked to try to identify 8 birds from pictures that were not shown during the study period, but which belong to one of the species you studied.



Once the study phase is over, click test to start the test and match each picture to a species name. There is no time limit on the test. Simply click next once you have selected each of your answers.

Based on previous research, one would predict that, in general, you would do better in the interleaved condition, where the species are mixed together in the study phase, than you would in the ‘massed,’ or grouped condition, where the pictures are presented in species groups. The question the researchers wanted to home in on in their first experiment was about the mechanism that made interleaved study more effective.

So, their experiment was conducted much like the one above, except with three groups, which all received the interleaved presentation. However, two of the groups were interrupted in their study by trivia questions in different ways. One group—the alternating trivia group—received a trivia question after every picture; the other group—the grouped trivia group—received 8 trivia questions after every group of 8 interleaved pictures. The third group—the contiguous group—received no interruption in their study.

What the researchers discovered is that while the contiguous group performed the best (of course), the grouped trivia group did not perform significantly worse, while the alternating trivia group did perform significantly worse than both the contiguous and grouped trivia groups. This was seen as providing some confirmation for the discriminative-contrast hypothesis:

Interleaved studying might facilitate noticing the differences that separate one category from another. In other words, perhaps interleaving is beneficial because it juxtaposes different categories, which then highlights differences across the categories and supports discrimination learning.

In the grouped trivia condition, participants were still able to take advantage of the interleaving effect because the disruptions (the trivia questions) had less of an effect when grouped in packs of 8. In the alternating trivia condition, however, a trivia question appeared after every picture, frustrating the discrimination mechanism that seems to help make the interleaving effect tick.

Takeaway Goodies (and Questions) for Instruction

The paper makes it clear that interleaving is not a slam dunk for instruction. Massed studying or practice might be more beneficial, for example, when the goal is to understand the similarities among the objects of study rather than the differences. Massed studying may also be preferred when the objects are ‘highly discriminable’ (easy to tell apart).

Yet, many of the misconceptions we deal with in mathematics education in particular can be seen as the result of dealing with objects of ‘low discriminability’ (objects that are hard to tell apart). In many cases, these objects really are hard to tell apart, and in others we simply make them hard through our sequencing. Consider some of the items listed in the NCTM’s wonderful 13 Rules That Expire, which students often misapply:

  • When multiplying by ten, just add a zero to the end of the number.
  • You cannot take a bigger number from a smaller number.
  • Addition and multiplication make numbers bigger.
  • You always divide the larger number by the smaller number.

In some sense, these are problematic because they are like the sparrows and finches above when presented only in groups—they are harder to stop because we don’t present them in situations that break the rules, or interleave them. Appending a zero to a number to multiply by 10 does work on counting numbers but not on decimals; addition and multiplication do make counting numbers bigger until they don’t always make fractions bigger; and you cannot take a bigger counting number from a smaller one and get a counting number. For that, you need integers.

Notice any similarities above? Can we please talk about how we keep kids trapped for too long in counting number land? I’ve got this marvelous study to show you which might provide some good reasons to interleave different number systems throughout students’ educations. It’s linked above, and below.

Images credits: All About Birds, Anne Davis 773