Readiness Assurance Process

In this phase you need to:

Select appropriate readings
Develop a list to topics to test
Develop a test plan
Write RAP questions
Get ready for the classroom

Optional Readings

Multiple-Choice Tests Exonerated, at Least of Some Charges: Fostering Test-Induced Learning and Avoiding Test-Induced Forgetting by Jeri L. Little, Elizabeth Bjork, Robert A. Bjork, and Genna Angello
Question Writing Manual by National Medical Board Examiners

Selecting the Right Reading

You need to identify what specific knowledge students will need to effectively engage with the 4S activities. This is not everything they need to solve every activity but what they require as an entry point to the problem-solving conversation. You do this by mapping back from the 4S application activity to important foundational knowledge that the students will need to be successful. When you are clear on the knowledge students need to know, you are then ready to select appropriate preparation materials.

Next, you need to select appropriate preparation materials. There is an iterative loop with the following step as you refine the concepts to be tested, and then select and refine the preparation materials. You can use a variety of preparatory materials including readings, videos, lecture recordings, or narrated PowerPoint’s.

Over the years we have discovered that less is more with readings. The amount of readings that students will tolerate depends on the particular discipline and institutional context. Our readings are closer to 25 pages for 2 weeks, which is down from our original 75 pages for two weeks. We found that students were spending a short, fixed amount of time completing readings without regard for complexity and length of readings. Remember the Readings and the Readiness Assurance Process is not trying to be comprehensive. It is just giving students an entry point to the problem-solving conversation.

One aside – when teachers are first introduced to the idea of the flipped classroom, they are often concerned on how to cram their 1 hour lectures into a 10-12 minute preparatory video. This is the wrong way to look at it. These short preparation materials are just to get students started. It is not all that students learn in a module, so 1 hour of lecture content is too much. Students will learn the additional content during the 4S team tasks.

Identifying Topics to Test

Once you have selected readings for the week’s preparation, skim the readings and make a list of critical ideas that student need to get from the preparatory materials. Often after skimming, you can do a slower read, listing important concepts, definitions, and ideas that the student need to get started. You use this list both to develop questions that check students understanding of critical concepts, principles, factual understanding, laws, rules, etc. and to develop preamble/wrapper for reading, so students know what to pay attention to.

The test should be a mix of approximately 20% remembering (did you do the readings?), approximately 60% understanding (did you understand what you read?), and finally, 20% application,. The application questions can be in the form of “which concept applies to this situation” (are you ready to use what you have read?). To use a book analogy, you want to write these tests more at the table-of-contents level then at the index level.

You can include a few simpler questions that just provide simple accountability that the student has completed the readings. Try to ask about topics that students are likely to interpret incorrectly. Test common misconceptions that might undermine students’ ability to successfully begin problem-solving. You can ask which concept applies to a given situation or scenario. You can focus on the relationship between concepts; this is an efficient way to test two concepts at once.

Writing Good Questions

Multiple-Choice questions have two main parts: the question stem or leader, and the options (which include a correct answer). When beginning to construct a multiple-choice question, write the stem of the question first. A well-constructed stem is a stand-alone question that could be answered without examining the options. The wording of the stem and the verbs it contains determines the overall difficulty of the question.

Bloom’s Taxonomy (lower levels)

Remembering

recalling, defining, recognizing, listing, describing, retrieving, naming

Common Question Leaders: How is…? Where is…? When did … happen? How would you describe…? Can you select….? Why did….?

Understanding

explaining ideas or concepts, interpreting, summarizing, paraphrasing, classifying, explaining, locating, identifying, restating

Common Question Leaders: How would you classify…? What facts or ideas show….? Interpret in your own words…? Which statement supports…? How would you summarize…? What is the main idea of…?

Applying

using information in another familiar situation, implementing, carrying out, using, executing , translate, employing, illustrating

Common Question Leaders: What is the best first step? What is the most significant problem? What would be the worst thing to do? Would it be a mistake to…? What is the most common mistake? Which test would you order next? What is the most common diagnosis? How would you use…? How would you solve? What is the most logical order? What approach would you use..? What would result if….? What facts would you select to show…?

This next section highlights some of Bill Roberson’s excellent work.

Writing Questions at different levels Low-level Questions

What did the text say? (Remembering)
What did the text mean? (Understanding)
How could you apply it? (Recognize an example of a concept)

Low-level Questions that invite discussion

Which statement is most accurate?
Based on the theory that you just read about, what is most likely to happen is we apply X?
Which of these items best represent the qualities/characteristics of X?

Higher-level Question that invite discussion

Based on what you have read about theory A, which of the strategies listed below has the best chance of success, given the specified conditions (X, Y, Z)?

Another Great Question Writing Framework (Gronlund)

ILLUSTRATIVE KNOWLEDGE QUESTIONS

Knowledge of Terminology

What word means the same as ________?
Which statement best defines the term ________?
In this sentence, what is the meaning of the word ________?

Knowledge of Specific Facts

Where would you find ________?
Who first discovered ________?
What is the name of ________?

Knowledge of Conventions

What is the correct form for ________?
Which statement indicates correct usage of ________?
Which of the following rules applies to ________?

Knowledge of Trends and Sequences

Which of the following best describes the trend of ________?
Which is the most important cause of ________?
Which of the following indicates the proper order of ________?

Knowledge of Classifications and Categories

What are the main types of ________?
What are the major classifications of ________?
What are the characteristics of ________?

Knowledge of Criteria

Which of the following is a criterion for judging ________?
What is the most important criterion for selecting ________?
What criteria are used to classify ________?

Knowledge of Methodology

What method is used for ________?
What is the best way to ________?
What would be the first step in making ________?

Knowledge of Principles and Generalizations

Which statement best expresses the principle of ________?
Which statement best summarizes the belief that ________?
Which of the following principles best explains ________?

Knowledge of Theories and Structures

Which statement is most consistent with the theory of ________?
Which of the following best describes the structure of ________?
What evidence best supports the theory of ________?

ILLUSTRATIVE COMPREHENSION AND APPLICATION QUESTIONS

Comprehension Questions

Which of the following is an example of ________?
What is the main thought expressed by ________?
What are the main differences between ________?
What are the common characteristics of ________?
Which of the following is another form of ________?
Which of the following best explains ________?
Which of the following best summarizes ________?
Which of the following best illustrates ________?
What do you predict would happen if ________?
What trend do you predict in ________?

Application Questions

Which of the following methods is best for ________?
What steps should be followed in applying ________?
Which situation would require the use of ________?
Which principle would be best for solving ________?
What procedure is best for improving ________?
What procedure is best for constructing ________?
What procedure is best for correcting ________?
Which of the following is the best plan for ________?
Which of the following provides the proper sequence for ________?
What is the most probable effect of ________?

From: How to make Achievement Tests and Assessments – 5^th edition by Norman Gronlund

Some Rules for Question Writing

For good question stems, consider following rules:

Stems should be stand-alone questions.
Stems should be grammatically complete.
Negative stems should be used with caution.
If a key word appears consistently in the options, try to move it to the stem.
Word the stem such that one option is indisputably correct.

For creating good options, consider following rules:

Make sure each incorrect option is plausible but clearly incorrect.
Make sure that the correct answer (keyed response) is clearly the best.
Avoid, if possible, using “all of the above”.
Use “none of the above” with caution.
Try to keep options similar lengths, since test-wise students will pick the longest option if unsure (too long to be wrong).
Make sure options are grammatically consistent with the stem (question leader) and use parallelism.
Make sure that numerical answers are placed in numerical order, either ascending or descending.

Well-constructed multiple-choice questions are not easy to create. But the quality of the multiple-choice questions you use in your Team Test can make or break the tone of your class. Nothing is more uncomfortable than rushing poor questions to the classroom and having to endure the inevitable student backlash. Good questions are absolutely essential to our success, and putting in the effort to write good questions is worth your time and attention.

Spend time reviewing and revising your questions. It can be very helpful to have a colleague look at your questions. When we write them we are often too close to see all the mistakes. Just like good writing is about good editing, good questions are about reflection and revision.

IF-ATs (aka Scratch Cards)

Immediate Feedback Assessment Technique

There back in production https://www.cognalearn.com/IFAT

A special kind of scoring sheet, known as an IF-AT form (Immediate Feedback Assessment Technique), is typically used for the team Readiness Assurance tests. IF-ATs are ”scratch-and-win”-style scoring sheets. They dramatically increase the quality of discussion in the tRAT process and, more importantly, provide rich, immediate corrective feedback. Students absolutely love using these test cards. You can expect high-fives and cheering as students complete the tRAT. We have even had some students thank us for the test! If you have not tried these, you must!

These “scratch-and-win”-style answer sheets were invented by Mike Epstein at Ryder University. On an IF-AT form, each question has a row of boxes that can be scratched like a lottery ticket. A small star, exposed by scratching one of the boxes, indicates the correct answer. Students must scratch off the opaque coating one box at a time in hopes of finding the right answer. The power of these cards is in the conversation that students are forced into as they try to generate shared understanding and consensus before choosing which box to scratch. The other powerful feature of these cards is the immediate, corrective feedback.

Decremental scoring is used to induce teams to continue to discuss the questions after an incorrect scratch. One possible point-allocation system for an IF-AT card could be four points for the right answer on the first scratch, two points for the right answer on the second scratch, one point for the right answer on the third scratch, and zero if they needed to scratch off all four boxes to reveal the correct answer. This encourages teams to engage with a question and continue the conversation until they know the correct answer.

Teachers new to TBL often ask how to print IF-ATs to match their tests. In fact, each set of numbered cards already has the stars in a particular pattern and it is the teacher who arranges the answer choices of the test to match the selected card. Cards are available in many different answer patterns to give the teacher flexibility in arranging answer choices and to prevent the students memorizing or predicting the answers. Each card has an identifying key number on a perforated tab at the bottom that can be removed by the teacher. ~~The IF-AT cards are available in lengths of 10, 25, and 50 questions and with either four options (A-D) or five options (A-E). IF-AT forms are available at www.epsteineducation.com~~

There have been some alternatives suggested on TBL listserv

“There are also some color-changing markers that work with invisible ink. You mark the right answer in invisible ink and the students use the marker and know it’s the right answer if the ink changes color. “

“I recently just made my own IF-AT scratch-off forms since Epstein wasn’t selling them anymore. My classes are small enough that even though it’s a process, it’s still feasible for me to do it (I have roughly 40 students per class). You can do this pretty easily in Microsoft word. Four cards fit on one 8.5×11 sheet of cardstock. You can’t see through colored cardstock (at least on the blue and green cardstock we had in our office supply area). Then you can order the scratch-off stickers from Amazon for about $20, cut them into strips, and stick them on the cards. I have a TA who does that part because even though it’s not difficult, it’s fairly time-consuming. This semester is my first time doing these cards and the students love it!”

“Not sure if folks are aware of the Group RAT site. This is a wonderful site that takes the concept of immediate feedback testing online. Wonderful site, super user friendly (once you learn how to set up your RATs in the right format) A hardy thanks to Gary Theilman for setting up this free site!”

“I used Qualtrics last year out of necessity during online teaching, and I decided to continue using Qualtrics (instead of scantron and IF-AT) this year by choice even though my university has returned to in-person teaching. The Qualtrics approach works great, and also saves paper“

“Here is a link with a new iRAT using Qualtrics video that reviews set-up and shows how to add up the scores in Excel.

https://www.loom.com/share/837689a202f54433a3f5da1827dc1bb4”

See IF-ATs in Action

Frequently Asked Questions

What are common student complaints about RATs?

Students will say “testing before teaching makes no sense” – you will hear it, if you don’t properly sell the idea of TBL and explicitly highlight the purpose of the Readiness Assurance Process. Which is low stakes tests that get students ready for the main course – learning how to apply the course content in the Application Activities.

Good student sometimes worry about lower grades on iRATs and voice these concerns. Our iRAT average is typically 65-70% and tRAT averages of 95-100%. Good students sometimes need to be reassured that the RAT’s don’t count for much, there focus is on learning, and the high team scores will pull up their RAT grades.

How often should I give RATs?

A common early mistake is to give too many RAP’s. Giving RAP’s tests every week is usually too often unless classes are long and only meet weekly. Typically we do a RAP every 2-3 weeks in a regular 13 week course. Remember the Readiness Assurance Process is just about getting students the minimum they need to begin problem solving. The students will learn plenty more during the Application Activities. In certain course configuration with long class meetings once a week, short readings and weekly RAP’s can make sense. Better to have too few, then too many – remember these aren’t about testing, there about getting your students ready for problem solving.

How much should the RATs count?

Not too much – If you assign a large grade to each RAP, you can turn it into high stakes testing. Assign just enough to make it worthwhile for students to do the readings and prepare. The typically range for a RAT is 2-5% for iRAT and 2-5% for tRAT. We have always used 1.5-2% for each RAT.

What kinds of questions are good questions for the RATs?

RAP questions are typically constructed at Bloom’s levels Remember, Understand and light Application. The RAP test highlight concepts more at a table of content level than index level. Remember these questions are just to get students ready for the activities that follow. These are not final summative tests of all that has been learned.

Next: Begin developing 4S activities