December 9, 2016

Design: How We Assess It


<< Previous: Content Next: Interpretation >>

Our Simulation Assessments

We assess the practices of eliciting and interpreting student thinking through the use of simulation assessments, in which preservice teachers interact with a “student” (i.e., someone trained to respond in standardized ways guided by a highly specified student thinking and interaction profile). Each assessment has three stages:


A preservice teacher analyzes a student’s written work on a mathematics problem and prepares to interact with the “student” about the problem



A preservice teacher interacts with a “student” about the written work



An assessment proctor interviews the preservice teacher about his or her interpretations of the student’s thinking

The three stages are elaborated below using one of our simulation assessments as an example.

Example: Long Division

Developing each simulation involves a number of design decisions. These include choices about:

  • the focal mathematical content
  • the correctness of the student’s answer (correct or incorrect)
  • the type of process being used (standard, invented, alternative)
  • the degree to which the student understands the process
  • the developmental/grade level of the student
  • the student’s “way of being” in the interaction

In the example below, we selected division using a standard long-division algorithm with a correct answer. We decided that the student would know the procedure for long division, but would think that the remainder can simply be written to the right of a decimal point. In other words, the student is not thinking of the 3 in the quotient as 3 tenths. We identified the student as a fifth-grade student who was not predisposed to elaboration or explaining her thinking. The numerical example chosen, in particular the divisor of 10, is key because the student’s process happens to yield the correct answer. In many situations this would not be the case.

Stage 1: Preparation

In the first stage of the assessment, preservice teachers are given a copy of the standardized student’s work on a problem and have 10 minutes to prepare for their upcoming interaction. In this division example, even though the student produces a correct answer, preservice teachers still need to determine how the student reasoned and what the student understands and does not understand. There is not much residue of the student’s understanding in the mathematical work. Students can produce correct answers using procedural knowledge without understanding the concepts behind the procedure. An important task of teaching is to probe and make sense of students’ mathematical processes and understanding, both when students have correct answers and when they do not. Preservice teachers are told that they should elicit and probe the standardized student’s thinking to understand the steps she took, why she performed particular steps, and her understanding of the key mathematical ideas involved.

Example of the preparation materials given to preservice teachers

Example of a preservice teacher’s preparation

In these preparation notes, the preservice teacher observed that the student’s answer is correct and that the student appears to know the steps for the procedure. The preservice teacher questioned whether the student understands those steps and concluded that it was not clear from the written work how the student “used the 3 to get the .3 part” or whether the student understands remainders. The potential questions generated include general questions about the problem and the steps used, as well as more specific questions and additional examples to probe the student’s understanding of the remainder.

Stage 2: Simulation

Preservice teachers have five minutes to interact with the “student” in the role of the teacher.

The “student” acts in accordance with a highly specified “student profile.” The profile details how the problem was completed and the thinking behind it. The profile articulates rules for reasoning and responding to the preservice teacher’s questions, and includes specific replies to commonly asked questions.

Student Profile

View excerpts from a student profile

A teacher educator (the “assessor”) observes and scores the assessment. The assessment is video recorded.

Below are two performances on the example task. For each performance, two videos are provided: (1) a record of the performance without commentary and (2) that same performance annotated with observations a teacher educator might make. These observations are tied to how we interpret the assessment in connection with the decomposition of the practice.

Simulation – Sample Preservice Teacher Performance A

Without annotations

With annotations

Simulation – Sample Preservice Teacher Performance B

Without annotations

With annotations

Stage 3: Interview

Following the interaction with the “student”, the assessor asks the preservice teacher a series of questions. These questions focus on the preservice teacher’s interpretation of the student’s process and understanding, as well as the preservice teacher’s mathematical knowledge for teaching relevant to the situation. Specifically, preservice teachers are asked to:

  • Describe the student’s process
  • Characterize the student’s understanding of the process, including underlying mathematical ideas
  • Generate a follow-up problem which could be used to confirm the student’s process
  • Anticipate how the student would solve a similar problem, if the student used the same process
  • Anticipate the student’s understanding of specific parts of the process, using evidence from the simulation interaction to support the anticipation
  • Characterize the generalizability of the student’s process
<< Previous: Content Next: Interpretation >>