Data Hounds

I used to have a beagle named Mabel who was probably the cutest dog I have ever seen but impossible to walk. Mabel was a data hound, and the instrument well-suited to her pursuit of information was her nose, which made every square inch of territory she encountered a moment of wonder. I observed that it was not the wonder of the world that kept her nose to the ground, however, for what caught her olfactory attention then became the need to reduce uncertainty about what she was smelling. The world is full of observable phenomena, and Mabel wanted to smell it all and figure out what it was.

Like Mabel, we are all data hounds, but unlike my little beagle, we use data to reduce uncertainty about our world in order to make better decisions. In fact, our decisions would be considerably worse if we did not use data in making them. To take my morning as an example, in the hope of illustrating my point but at the risk of being tedious, I woke up and measured how long I had until I had to get up in order to be at school on time. Along with that measurement was the measurement of my wakefulness. While taking those two measurements, I made a quick measurement of my day and quickly decided that I needed to get up. First stop was the shower, where I turned on the water and measured its temperature, made an adjustment, measured it again, and adjusted again. At breakfast, I made coffee, and measured its temperature before drinking. It was too hot, so I decided to let it cool a bit. I measured it again, and decided it was cool enough to drink but hot enough to have that deep coffee flavor we coffee addicts love. I checked the newspaper for the weather, for I needed a measurement of the day’s heat and precipitation. Now strictly speaking, those data were forecasts and not observed phenomena, but weather forecasts are based on observed phenomena and use statistical models to forecast what meteorological phenomena I would encounter that day. Rain later and a high of 62. Now the paper lists weather conditions all over the world, but I limited the data I used in making my fashion decisions to Chicago. I could see stars in the early morning sky, but I trust Tom Skilling so I grabbed an appropriate coat and an umbrella. Measuring the time once again, I left for school. Without data to reduce my uncertainty about the temperature of the shower, temperature of the coffee, weather, and time, I might have frozen or parboiled in the shower, scalded my tongue, got soaked, froze, and been late. Or perhaps I would have avoided all those unfortunate events by guessing, but why take the chance when the world is full of data the measurement of which allows me to improve my decision making?

All that data measurement to reduce uncertainty and improve decision making in 30 minutes. You do the same thing, all day long, every day.

I must admit, I am a data hound when it comes to educating children. To function, schools have operations, and those operations produce effects. Students attend class and schools take attendance. Schools instruct and students learn. Programs are offered and parents benefit. If anything that takes place in a school makes a difference, that difference is an effect that can be measured, and I want to know what difference our efforts as a school are making in the education of children. To do that, we measure effects, but not just any effects. We do not have the resources (time, budget, personnel) to measure everything, and in fact we do not need to measure anything. For if the question is what difference we are making in our school operations, we should be looking at those differences that make the biggest difference in the education of children. The government has chosen to measure once a year what students have learned, on one test, using, as we say, one modality. We can do better than that. We can measure how much students are learning along the way in several ways, and we can measure how well we are operating as a school, and we can make decisions based on the data we get from those measurements. There is much uncertainty in school operations: how much are the students learning? How effective is our teaching? How well are we communicating? How satisfied are our parents with our programs and services? We can observe the difference our operations make, measure that difference, and reduce our uncertainty about how well we are doing as a school. Data hounds.

How do we get data from our school operations? At CVCS, I believe the most important school data reveal what students have learned. We can reduce our uncertainty about what they have learned by observing their progress in the OLS or LMS, but we can also measure what they have learned in smaller chunks, which allows us to target our instruction at any area of deficiency. That’s why K-8 students take Achievement Exercises every week. I have been working with teachers to articulate in a rubric what we believe effective virtual and face-to-face instruction is, and together we are working to improve our instructional effectiveness. The relationship between the data on the quality of instruction and gains in student learning is not necessarily direct, but the instructional rubric allows us to reduce our uncertainty as a school about our collective instructional effectiveness and make inferences about its effect on student learning.

As for how parents perceive our efforts, we could wait to see how many families stay at CVCS, but that measurement lags behind the conditions that led to it. To get at families’ perceptions about our programs and services, I will be using random surveys. That means I will use either a table of random numbers or a web site like randomizer.org to randomly select Learning Coaches to call and ask a few questions. I can get information from focus groups, and I like SurveyMonkey, but neither of those instruments is as accurate for understanding the population as a random selection. A group of people who choose to answer a survey is not a random sample. A focus group could be a random sample if participants are randomly selected, but a group of people discussing an issue is just a group of people, not a sample I can use to generalize onto the population. And there is lots of power in random samples. If you are making soup, it is unnecessary to eat the entire pot in order to know if it needs more salt. One spoonful will tell you that, provided that that spoonful has all the ingredients distributed in it that the whole pot has.

This desire to know is more a moral imperative than a curiosity about statistics. Schools stand in a relationship with children’s individual pursuit of happiness, for schools can operate in a way that empowers students in their pursuit of happiness. I am not suggesting that the sum total of a student’s future happiness is tied to the work of school. I would argue, however, that in 2009, students’ ability to flourish will be partly determined by their success in school, and the condition of our civil society will rest in the hands of students who currently work together in schools. Thus the desire to reduce our uncertainty about how our students are learning and how we are doing as a school answers a moral imperative to become the best school we can be.

Postscript: After a sunny start the rain indeed came on my way home. Tom Skilling was right.