Retrieval Practice with Kindle: Feel the Learn

I use Amazon’s free Kindle Reader for all of my (online and offline) book reading, except for any book that I really want that just can’t be had digitally. Besides notes and highlights, the Reader has a nifty little Flashcards feature that works really well for retrieval practice. Here’s how I do retrieval practice with Kindle.

Step 1: Construct the Empty Flashcard Decks

Currently I’m working through Sarah Guido and Andreas Müller’s book Introduction to Machine Learning with Python. I skimmed the chapters before starting and decided that the authors’ breakdown by chapter was pretty good—not too long and not too short. So, I made a flashcard deck for each chapter in the book, as shown at the right. On your Kindle Reader, click on the stacked cards icon. Then click on the large + sign next to “Flashcards” to create and name each new deck.

Depending on your situation, you may not have a choice in how you break things down. But I think it’s good advice to set up the decks—however far in advance you want—before you start reading.

So, if I were assigned to read the first half of Chapter 2 for a class, I would create a flashcard deck for the first half of Chapter 2 before I started reading. And, although I didn’t set titles in this example, it’s probably a good idea to give the flashcard deck a title related to what it’s about (e.g., Supervised Learning).

You still need to read and comprehend the content. Retrieval practice adds, it doesn’t replace. So, I read and highlight and write notes like I normally would. I don’t worry at this point about the flashcards, about what is important or not. I just read for the pleasure of finding things out. I highlight things that strike me as especially interesting and write notes with questions, or comments I want to make on the text.

Read a section of the content represented by one flashcard deck. Since I divided my decks by chapter, I read the first chapter straight through, highlighting and making notes as I went.

The reading doesn’t have to be done in one sitting. The important thing is to just focus on reading one section before moving on to the next step.

Step 3: Create the Fronts for the Flashcards

Now, go through the content of your first section of reading and identify important concepts, items worth remembering, things you want to be able to produce. You’ll want to add these as prompts on your flashcards. You don’t necessarily have to write these all down in a list. You can enter a prompt on a flashcard, return to the text for another prompt, enter a prompt on another flashcard, and on and on.

Screenshot 1

Screenshot 2

Screenshot 3

When you have at least one prompt, click on the flashcard deck and then click on Add a Card (Screenshot 1) and enter the prompt.

Enter the prompt at the top. (Screenshot 2) This will be the front of the flashcard you will see when testing yourself. Leave the back blank for the moment. Click Save and Add Another Card at the bottom right to repeat this with more prompts.

When you are finished entering one card or all the cards, click on Save at the top right. This will automatically take you to the testing mode (Screenshot 3), which you’ll want to ignore for a while. Click on the stacked cards icon to return to the text for more prompts. When you come back to the flashcards, your decks may have shifted, since the most recently edited deck will be at the top.

Importantly, though, Screenshot 3 is the screen you will see when you return and click on a deck. To add more cards from this screen, click on the + sign at the bottom right. When you are done entering the cards for a section, get ready for the retrieval practice challenge! This is where it gets good (for learning).

Step 4: Create the Backs for the Flashcards

Rather than simply enter the backs of the flashcards from the information in the book, I first fill out the backs by simply trying to retrieve what I can remember. For example, for the prompt, “Write the code for the Iris model, using K Nearest Neighbors,” I wrote something like this on the back of the card:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris_dataset = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris_dataset.data, iris_dataset.target)

There are a lot of omissions here and some errors, and I moved things around after I wrote them down, but I tried as hard as I could to remember the code. To make the back of the card right, I filled in the omissions and corrected the errors. As I went through this process with all the cards in a section, I edited the fronts and backs of the cards and even added new cards as the importance of some material presented itself more clearly.

Create the backs of the flashcards for a section by first trying as hard as you can to retrieve the information asked for in the prompt. Then, correct the information and fill in omissions. Repeat this for each card in the deck.

Step 5: Test Yourself and Feel the Learn

One thing you should notice when you do this is that it hurts. And it should. In my view, the prompts should not be easy to answer. Another prompt I have for a different chapter is “Explain how k-neighbors regression works for both 1 neighbor and multiple neighbors.” My expectations for my response are high—I want to answer completely with several details from the text, not just a mooshy general answer. I keep the number of cards per chapter fairly low (about 5 to 10 cards per 100 pages). But your goals for retaining information may be different.

But once you have a set of cards for a section, come back to them occasionally and complete a round of testing for the section. To test yourself, click on the deck and respond to the first prompt you see without looking at the answer. Try to be as complete (correct) as possible before looking at the correct response.

To view the correct response, click on the card. Then, click on the checkmark if you completely nailed the response. Anything short of that, I click on the red X.

For large decks, you may want to restudy those items you got incorrect. In that case, you can click on Study Incorrect to go back over just those cards you got wrong. There is also an option to shuffle the deck (at the bottom left), which you should make use of if the content of the cards build on each other, making them too predictable.

Contiguity Effective for Deductive Inference

research post

The discourse that surrounds the technicalities in this paper contains an agenda: to convince readers that the benefits of retrieval practice extend beyond the boring old “helps you remember stuff” caricature to something more “higher order” like deductive inference. But I’m not convinced that the experiments show this. Rather, what they demonstrate fairly convincingly is that informational contiguity, not retrieval practice, benefits inference-making. A related result from the research, on the benefits of text coherence, is explained here.

The Setup

The arch-nemesis of this research is a paper by Tran, et al. last year, which appeared to show some domain limitations on retrieval practice:

They found that retrieval practice failed to benefit participants’ later ability to make accurate deductions from previously retrieved information. In their study, participants were presented sentences one at a time to learn . . . The sentences could be related to one another to derive inferences about particular scenarios. Although retrieval practice was shown to improve memory of the sentences relative to a restudy control condition, there was no benefit on a final inference test that required integration of information from across multiple sentences.

So, in this study, researchers replicated Tran et al.’s methods, except in one important way: they did not present the sentences to be learned one at a time but together instead.

Participants were each presented with four scenarios (two of which are outlined at right) consisting of seven to nine premises in the form of sentences. In each scenario, deductions to specific conclusions were possible. For two of the four scenarios, subjects used retrieval practice. They were given a chance to read the sentences in a scenario at their own pace and then were shown the sentences again for five minutes—in cycles where the order of the sentences was randomized. During this five-minute session, subjects were asked to type in the missing words in each premise (between one and three missing words). The complete sentences were then shown as feedback. Each participant used restudy for the other two scenarios. During the restudy five-minute session, subjects simply reread the premises again, in cycles again, with the order of the premises randomized for each cycle.

The Results and Discussion

Two days later, participants were given a 32-item multiple choice test which “assessed participants’ ability to draw logical conclusions derived from at least two premises within each scenario.” And consistent with the researchers’ hypothesis, the retrieval practice conditions yielded significantly better results on a test of deductive inference than did the restudy conditions.

Yet, it’s not at all clear that retrieval practice was the cause of the better performance with respect to inference-making. There was another cause preceding it: the improved contiguity of the presented information, as compared with Tran et al.’s one-at-a-time procedure. It’s possible that the effectiveness of retrieval practice is limited to recall of already-integrated information, and the contiguity of the premises in this study allowed for such integration, which, in turn, allowed retrieval practice to outperform restudy. It is a possibility the researchers raise in the paper and one that, in my view, the current research has not effectively answered:

However, other recent studies have failed to find a benefit of retrieval practice for learning educational materials (Leahy et al. 2015; Tran et al. 2015; Van Gog and Sweller 2015). These studies all used learning materials that required learners to simultaneously relate multiple elements of the materials during study and/or test. Such materials that are high in element interactivity need constituent elements to be related to one another in order for successful learning or task completion (element interactivity may also be considered as a measure of the complexity of materials, see (Sweller 2010)).

What we can say, with some confidence, is that even if the benefits of retrieval practice were limited to improvements in recall (as prior research has demonstrated), such improvements do not stand in the way of improvements to higher-order reasoning, such as inference-making. (And shaping the path for students, such as improving informational contiguity can have a positive effect too.)

Eglington, L., & Kang, S. (2016). Retrieval Practice Benefits Deductive Inference Educational Psychology Review DOI: 10.1007/s10648-016-9386-y