Is Your State Adding Value?
03/25/2010
A+
|
a-

Americans have devoted an enormous amount of effort over the centuries to devising useful baseball statistics. In recent years, Americans have talked a lot about devising useful educational statistics.

For example, I've pointed out a million times over the last decade that it doesn't make much sense to judge teachers, schools, or colleges by their students' test scores. Most of the time, all you are doing is determining which kids were smarter to start with. Logically, it makes more sense to judge their "value added" by comparing how the students score now to how they scored in the past before the people or institutions being measured got their mitts on the students.

Over the last few years, everybody who is anybody in education — Bill Gates, Arne Duncan, you name it — has come around to this perspective (although they won't use the word "smarter").

A big problem, however, is that this value added idea remains almost wholly theoretical because almost none of the prominent educational statistics are published in value added form.

In contrast, when Bill James was pointing out 30 years ago that Batting Average, traditionally the most prestigious hitting statistic (the guy with the highest BA was crowned "Batting Champion"), wasn't as good a measure of hitting contribution as Slugging Average plus On-Base Percentage, he could show you what he meant using real numbers that were available to everybody, even if you had to calculate them yourself from other, more widely published statistics.

Readers would say, "Yeah, he's right. For example, Matty Alou (career batting average .307, but slugging average .381 and on-base percentage .345) wasn't anywhere near as good as Mickey Mantle (career batting average only .298, but slugging average .557 and on-base percentage .421). If you add on-base percentage and slugging average together to get "OPS," then Mickey had a .977 while Matty only had .726. And that sounds about right. Mickey was awesome, but it didn't always show up in his traditional statistics. Now, we've finally got a statistic that matches up with what we all could see from watching lots of Yankee games."

On the other hand, other innovative baseball statistics from that era have faded because they didn't seem to work as well in practice as in theory. Readers would be rightly skeptical that Glenn Hubbard and Roy Smalley Jr. really were all time greats, as these complicated formulas said they were.

A couple of years ago, Audacious Epigone and I stumbled upon a potentially promising fluke in the federal National Assessment of Educational Progress test scores by state. Since these tests are given every two years to representative samples of fourth and eighth graders, then you ought to be able to roughly estimate how much value the public schools in each state have added from 4th grade to 8th grade by comparing, say, a state's 2009 8th grade scores to that state's 2005 4th grade scores.

Granted, people move in and out of states, but if you just look at the scores for non-Hispanic whites, you can cut down the effect of demographic change to what might be a manageable level.

So, how to display this data in a semi-usable form? In the following table, I've put the Rank of each state. For example, in NAEP 4th Grade Reading scores in 2005, white public school students in Alabama ranked 48th (out of 52 — the 50 states plus D.C. and the Department of Defense schools for the children of military personnel). By 2009, this cohort of Alabamans was up to 47th in 8th Grade Reading. That's a Change in Rank of +1. Woo-hoo!

In contrast, in Math, Alabama's 4th Graders were 50th in 2005 and the state's 8th Graders were 50th in 2009, so that's a Change in Rank for Math of zero.

There are measures that are better for some purposes than Rank, but, admit it, ranking all the states is more interesting than using standard deviations or whatever.

A new idea is embodied in the last column, which reports the Difference in Rank between Math and Reading scores for 8th Graders in 2009. Because Alabama was 47th in Reading in 2009, but only 50th in Math in 2009, it gets a Difference in Rank of -3. Boo-hoo ...

What's the point of this last measure?

There's a fair amount of evidence that schools have more impact on Math performance than Reading performance. For example, math scores on a variety of tests have gone up some since hitting rock bottom during the Seventies (in most of America outside of Berkeley, the Seventies were when the Sixties actually happened). In contrast, reading and verbal scores have staggered around despite a huge amount of effort to raise them.

Why have math scores proven more open to improvement by schools than reading scores? One reason probably is that because kids only spend about 1/5th of their waking hours in school. And almost nobody does math outside of school, but some kids read outside of school. So, if you, say, double the amount of time spent in school on math, then you are increasing the total amount of time kids are spending doing math by about 98%. But if you double the amount of time spent on reading in school, there are some rotten stinker kids who read for fun in their free time, and thus you aren't doing much for them in terms of total hours devoted to reading.

Not surprisingly, a decade of the No Child Left Behind act, which tells states to hammer on math and reading and don't worry about that arty stuff like history and science, has seen continued slow improvements in math, but not much in reading — except at the bottom (i.e., the kids who don't read outside school).

So, by 8th grade, Reading scores would likely be a rough measure of IQ crossed with bookishness (personality and culture). In contrast, 8th Grade Math scores are more amenable to alteration by schools since kids aren't waiting in line to buy Harry Potter and the Lowest Common Denominator. So, the idea behind the final column is to compare rank on 8th Grade Math to rank on 8th Grade Reading. A positive number means your state has a better (lower) rank on Math than on Reading, which might reflect relatively well on your public schools given the raw materials it has to work with relative to other states.

For example, on the NAEP, Texas ranks 11th among white 8th graders in Reading, which is pretty good for such a huge state. But, it ranks a very impressive 4th among white 8th graders in Math, for a Difference in Ranking score of +7. This suggests Texas is doing something with math that's worth checking into. Maybe they are just teaching to the test, but this is the NAEP, which isn't a high-stakes test. And there are worse things than teaching to the test. (Whatever they are doing, they are starting young, because Texas ranks 2nd in Math for white 4th Graders.)

So, here is this huge table:

 NAEP Read Read Read Chg Math Math Math Chg Math-Read Public 4th 8th 4th-8th 4th 8th 4th-8th 8th-8th White 2005 2009 09-05 2005 2009 09-05 09-09 Rank Rank Chg in Rnk Rank Rank Chg in Rnk Dif in Rnk Alabama 48 47 +1 50 50 +0 -3 Alaska 37 31 +6 31 21 +10 +10 Arizona 41 29 +12 36 27 +9 +2 Arkansas 34 46 -12 37 44 -7 +2 California 32 33 -1 25 36 -11 -3 Colorado 9 9 0 13 6 7 +3 Connecticut 4 2 +2 8 7 +1 -5 Delaware 3 14 -11 11 17 -6 -3 DC 1 +1 1 +1 0 DoDEA 8 5 +3 21 16 +5 -11 Florida 16 21 -5 14 37 -23 -16 Georgia 27 38 -11 33 34 -1 +4 Hawaii 40 45 -5 40 48 -8 -3 Idaho 30 35 -5 29 26 3 +9 Illinois 13 10 +3 28 18 +10 -8 Indiana 43 34 +9 26 29 +-3 +5 Iowa 42 41 +1 39 41 +-2 0 Kansas 33 19 +14 10 15 +-5 +4 Kentucky 46 37 +9 51 49 +2 -12 Louisiana 45 51 -6 41 45 -4 +6 Maine 36 39 -3 42 39 3 0 Maryland 7 3 +4 7 2 +5 +1 Massachusetts 2 4 -2 3 1 2 +3 Michigan 28 40 -12 22 42 -20 -2 Minnesota 12 7 +5 4 5 +-1 +2 Mississippi 49 48 +1 48 51 +-3 -3 Missouri 26 27 -1 45 32 13 -5 Montana 21 16 +5 35 10 +25 +6 Nebraska 18 20 -2 30 28 2 -8 Nevada 51 49 +2 44 40 +4 +9 New Hampshire 19 24 -5 20 23 -3 +1 New Jersey 6 1 +5 5 3 +2 -2 New Mexico 35 25 +10 49 38 +11 -13 New York 10 8 +2 16 19 +-3 -11 North Carolina 22 28 -6 6 8 -2 +20 North Dakota 20 22 -2 24 9 15 +13 Ohio 14 12 +2 12 30 +-18 -18 Oklahoma 50 50 0 46 46 0 +4 Oregon 44 36 +8 34 31 +3 +5 Pennsylvania 15 6 +9 17 14 +3 -8 Rhode Island 39 43 -4 43 43 0 0 South Carolina 38 44 -6 9 24 -15 +20 South Dakota 29 13 +16 23 12 +11 +1 Tennessee 47 42 +5 47 47 +0 -5 Texas 11 11 0 2 4 -2 +7 Utah 31 30 +1 38 33 +5 -3 Vermont 24 18 +6 32 22 +10 -4 Virginia 5 17 -12 15 20 -5 -3 Washington 17 15 +2 19 11 +8 +4 West Virginia 52 52 0 52 52 0 0 Wisconsin 23 26 -3 18 13 5 +13 Wyoming 25 32 -7 27 35 -8 -3
 NAEP Read Read Read Chg Math Math Math Chg Math-Read Public 4th 8th 4th-8th 4th 8th 4th-8th 8th-8th White 2005 2009 09-05 2005 2009 09-05 09-09 Rank Rank Chg in Rnk Rank Rank Chg in Rnk Dif in Rnk

As J.K. Simmons asks at the end of Burn After Reading, "What did we learn?"

I'm not terribly sure, either. Who knows enough about what goes on within the educational establishments of all the states to know whether these numbers make sense?

But, at least we have some value added numbers and aren't just still talking about how valuable they'd be if we ever got around to getting any.