‘Getting It Wrong’: Scantron Achievement Exercises as Best Practice

‘Getting It Wrong’: Scantron Achievement Exercises as Best Practice

In education, a “best practice” is the belief that one activity or method produces a better outcome than alternatives. As educators, we crave best practices because it allows us to place better bets for optimizing student learning. When we began the practice of holding weekly math exercises to see what students knew, I thought it was a best practice because those assessments are based on grade level state standards and allow us to pinpoint our teaching to what students did not know. Spreading out the standards across the year allows us to monitor student math attainment in grades 2-8 in fine detail and to remediate where necessary on very specific skills. In some cases, students had already learned the material, in others they had just learned it, and in still others students had never seen the material before, but every Monday, students in grades 2-8 have the opportunity to show us what they know in math.

At least two parents have asked whether it makes sense to assess students on math concepts and skills they have never seen and then teach those concepts and skills after the assessment. Intuitively, their concern makes sense, because students typically do poorly on material they have not been taught, especially skilled subjects like math. In fact, most of us learned this way: the teacher instructs the class, the students study what they are supposed to learn, and then the teacher gives some kind of assessment to determine what learning has taken place. Our intuition about that process being the right one probably comes from our experience in school.

Recently published research in Journal of Experimental Psychology: Learning, Memory, and Cognition and Journal of Experimental Psychology: Applied lead us to a startling, counterintuitive conclusion: students actually learn more if they are assessed on material before they are taught the material. When I read about this research in the October 20 edition of Scientific American (https://www.scientificamerican.com/article.cfm?id=getting-it-wrong), I had to read what it implies twice, so I will write it again in case you think that you misread something or that I mistyped it, this time quoting from the article: “Learning becomes better if conditions are arranged so students make errors.” For us, that means that students who are behind in math and have to fight their way through weekly math exercises and then learn the concepts learn more than if they had studied for the exercises ahead of time. To put it another way, from the perspective of this research in experimental psychology, Scantron weekly math exercises are a best practice for those who are behind in math.

The claim that one practice is a “best practice” is not a claim to absolute truth, and certainly not the end of the discussion of how best to educate students. Rather, it is a belief that must be continually tested. We rarely have the kind of conditions that would allow us to test our practices scientifically, but if we measure the outcomes we are getting, we can at least reduce the uncertainty about the effectiveness of our educational practices. One measurement of the effectiveness of the Scantron math exercises is the attainment of students over time. I will be reporting on that measurement, and others, in the presentation of the dashboard at the November Board meeting.

Let's Talk

Using blogs to communicate to a school community is new to me. I wrote guest columns for the local newspaper in the town where I used to work, I have written columns for newsletters, and I have used blogs and discussion boards in my university teaching. Last summer was the first time I put up a blog for a school community. I have learned a lot since then.

Here’s how I came to blogging for the school community. Last June, a couple parents at a Board meeting said they wanted to see the school’s strategy map and have the opportunity to comment on it. Someone from the Board suggested I put up a blog. So I did. Then I put up the narrative that explained what we meant by strategic objectives like “Improve Communication” and “Improve the Quality and Depth of Instruction.” After that, it occurred to me that blogging was a way of providing easy access to the school community to what I was thinking, on what as a school we are working, etc. Instead of writing a newsletter article, I would write a blog. As a result, I began using blogs to share my thoughts on school issues, books I was reading, and the importance of data. I still see blogs as the occasion to share what I am thinking because as the leader of the school what I am thinking will have an effect on how the school goes, and I want you to know what’s on my mind when it comes to the school. Those kinds of blogs will continue. But a blog has one dimension newsletters do not have: the opportunity to engage in a conversation.

So this blog and future blogs have another purpose, which I learned from the parent handbook I put up once a draft of it was complete. That blog aimed to give parents the opportunity to comment on the handbook draft so we could consider their ideas and suggestions as we approached a final draft. The comments poured in. I made the decision at that time to read the comments, share them with my team, discuss parents’ concerns and suggestions, but not respond to any posting. I thought that responding would take too much time because if I responded to one person, I would have to respond to everyone, and that could become my full time job. Was I wrong. Not responding to anyone online was a mistake, and I want to fix that.

That is why this blog and all future blogs are an invitation to you to write anything that is on your mind. I will respond to as many questions as I can, and responding to all questions is my goal. Moreover, I plan to blog on issues that have come up here, like the weekly Achievement Exercises in grades 2-8 and high school late policy. I look forward to engaging you in dialogue on what the school is doing to empower students to flourish.

In addition, I have created a Twitter account, which you can follow by going to http://twitter.com/cvcshos. With Twitter I can send out quick questions and advise you in 140 characters of an issue going on at the school at a given moment. And if you have a Twitter account you can respond, which is another way to get your feedback. I just posted last week’s attendance rates—and it was good news.

Twitter is completely new to me, and like any new tool, I will need to use it a while to learn how to use it effectively. And blogging as a way to engage parents is also new. The important thing is that we open up as many lines of communication as possible. Let’s talk.

Data Hounds

I used to have a beagle named Mabel who was probably the cutest dog I have ever seen but impossible to walk. Mabel was a data hound, and the instrument well-suited to her pursuit of information was her nose, which made every square inch of territory she encountered a moment of wonder. I observed that it was not the wonder of the world that kept her nose to the ground, however, for what caught her olfactory attention then became the need to reduce uncertainty about what she was smelling. The world is full of observable phenomena, and Mabel wanted to smell it all and figure out what it was.

Like Mabel, we are all data hounds, but unlike my little beagle, we use data to reduce uncertainty about our world in order to make better decisions. In fact, our decisions would be considerably worse if we did not use data in making them. To take my morning as an example, in the hope of illustrating my point but at the risk of being tedious, I woke up and measured how long I had until I had to get up in order to be at school on time. Along with that measurement was the measurement of my wakefulness. While taking those two measurements, I made a quick measurement of my day and quickly decided that I needed to get up. First stop was the shower, where I turned on the water and measured its temperature, made an adjustment, measured it again, and adjusted again. At breakfast, I made coffee, and measured its temperature before drinking. It was too hot, so I decided to let it cool a bit. I measured it again, and decided it was cool enough to drink but hot enough to have that deep coffee flavor we coffee addicts love. I checked the newspaper for the weather, for I needed a measurement of the day’s heat and precipitation. Now strictly speaking, those data were forecasts and not observed phenomena, but weather forecasts are based on observed phenomena and use statistical models to forecast what meteorological phenomena I would encounter that day. Rain later and a high of 62. Now the paper lists weather conditions all over the world, but I limited the data I used in making my fashion decisions to Chicago. I could see stars in the early morning sky, but I trust Tom Skilling so I grabbed an appropriate coat and an umbrella. Measuring the time once again, I left for school. Without data to reduce my uncertainty about the temperature of the shower, temperature of the coffee, weather, and time, I might have frozen or parboiled in the shower, scalded my tongue, got soaked, froze, and been late. Or perhaps I would have avoided all those unfortunate events by guessing, but why take the chance when the world is full of data the measurement of which allows me to improve my decision making?

All that data measurement to reduce uncertainty and improve decision making in 30 minutes. You do the same thing, all day long, every day.

I must admit, I am a data hound when it comes to educating children. To function, schools have operations, and those operations produce effects. Students attend class and schools take attendance. Schools instruct and students learn. Programs are offered and parents benefit. If anything that takes place in a school makes a difference, that difference is an effect that can be measured, and I want to know what difference our efforts as a school are making in the education of children. To do that, we measure effects, but not just any effects. We do not have the resources (time, budget, personnel) to measure everything, and in fact we do not need to measure anything. For if the question is what difference we are making in our school operations, we should be looking at those differences that make the biggest difference in the education of children. The government has chosen to measure once a year what students have learned, on one test, using, as we say, one modality. We can do better than that. We can measure how much students are learning along the way in several ways, and we can measure how well we are operating as a school, and we can make decisions based on the data we get from those measurements. There is much uncertainty in school operations: how much are the students learning? How effective is our teaching? How well are we communicating? How satisfied are our parents with our programs and services? We can observe the difference our operations make, measure that difference, and reduce our uncertainty about how well we are doing as a school. Data hounds.

How do we get data from our school operations? At CVCS, I believe the most important school data reveal what students have learned. We can reduce our uncertainty about what they have learned by observing their progress in the OLS or LMS, but we can also measure what they have learned in smaller chunks, which allows us to target our instruction at any area of deficiency. That’s why K-8 students take Achievement Exercises every week. I have been working with teachers to articulate in a rubric what we believe effective virtual and face-to-face instruction is, and together we are working to improve our instructional effectiveness. The relationship between the data on the quality of instruction and gains in student learning is not necessarily direct, but the instructional rubric allows us to reduce our uncertainty as a school about our collective instructional effectiveness and make inferences about its effect on student learning.

As for how parents perceive our efforts, we could wait to see how many families stay at CVCS, but that measurement lags behind the conditions that led to it. To get at families’ perceptions about our programs and services, I will be using random surveys. That means I will use either a table of random numbers or a web site like randomizer.org to randomly select Learning Coaches to call and ask a few questions. I can get information from focus groups, and I like SurveyMonkey, but neither of those instruments is as accurate for understanding the population as a random selection. A group of people who choose to answer a survey is not a random sample. A focus group could be a random sample if participants are randomly selected, but a group of people discussing an issue is just a group of people, not a sample I can use to generalize onto the population. And there is lots of power in random samples. If you are making soup, it is unnecessary to eat the entire pot in order to know if it needs more salt. One spoonful will tell you that, provided that that spoonful has all the ingredients distributed in it that the whole pot has.

This desire to know is more a moral imperative than a curiosity about statistics. Schools stand in a relationship with children’s individual pursuit of happiness, for schools can operate in a way that empowers students in their pursuit of happiness. I am not suggesting that the sum total of a student’s future happiness is tied to the work of school. I would argue, however, that in 2009, students’ ability to flourish will be partly determined by their success in school, and the condition of our civil society will rest in the hands of students who currently work together in schools. Thus the desire to reduce our uncertainty about how our students are learning and how we are doing as a school answers a moral imperative to become the best school we can be.

Postscript: After a sunny start the rain indeed came on my way home. Tom Skilling was right.