From the New Yorker:

The $1.8-Billion Lawsuit Over a Teacher Test

In the nineties, New York began requiring aspiring educators to take an exam. Thousands of people later claimed that the test was racially biased.

By Emma Green
October 31, 2023

… But, in 1991, a new law went into effect that meant that Wilds-Bethea, along with other teachers across the city, had to pass yet another test: the National Teacher Examination, or N.T.E. The N.T.E., which had been administered in other states for decades, was adopted by New York after several task forces concluded that low standards for teachers were hurting student performance, and called for greater knowledge of the liberal arts among educators. The exam covered basic knowledge of social studies, science, math, literature, and writing. New York City teachers—even those who had already been tenured and previously licensed—needed to pass in order to keep their full-time jobs with seniority and benefits. …

Wilds-Bethea failed the N.T.E. ten times. In 1993, the state began phasing out the N.T.E. and introduced an alternative exam, the Liberal Arts and Science Test, or last, which Wilds-Bethea took and failed three times. …

From 1993 through 1994, an average of eighty-four per cent of white test-takers passed the N.T.E., compared with forty-four per cent of Black test-takers, and forty per cent of Latino test-takers. From March, 1993, to June, 1995, the average pass rate for the last was ninety-three per cent for white test-takers, compared with fifty-three per cent and fifty per cent for Black and Latino test-takers, respectively. …

Those are pretty standard racial gaps, maybe slightly higher, but in the usual range found in almost all metrics, whether tests or job performance, with a sizable cognitive component.

Their class-action lawsuit, filed in 1996, was made up of Black and Latino teachers who failed the N.T.E. or the last. The case has dragged on for almost three decades, and has come to include more than five thousand plaintiffs. …

The court found that Black and Latino teachers clearly passed these tests at lower rates than white teachers. In order to prove that this wasn’t illegal, the defendants had to show that the test actually demonstrated what it promised: that teachers who did well on the test would do better in their jobs. A district-court judge, Constance Baker Motley—a Black woman and famous civil-rights lawyer who worked with Thurgood Marshall on Brown v. Board—initially found in favor of New York, concluding that the N.T.E. and the last were sufficiently job-related to justify their use. But the Second Circuit vacated part of that decision and dropped New York State as a defendant. Another district-court judge, Kimba Wood, subsequently found that the city had violated Title VII because the last was not properly validated, or proven to show what it said it showed. In 2021, the city agreed to a schedule of payments. This fight over tests has proved expensive: the city now owes many of these teachers significant back pay and other financial compensation. The payouts are expected to total about $1.8 billion—the highest dollar-value judgment ever brought against New York City. …

Individual plaintiffs are being awarded up to $2 million.

Actual question examples from the last and the N.T.E. are difficult to find, in part because portions of the trial transcript have been sealed—test-makers didn’t want their proprietary materials being made public. But in the available sections of the transcript, the prosecution’s expert witness, Frank Landy, a professor of industrial psychology at Penn State, did discuss a few N.T.E. questions that he found particularly problematic. One apparently asked test-takers to look at a performance stage and speculate about why its designer had created it in such an “unadorned” way. Another concerned the splitting of the hemispheres of the brain, which Landy said would require a sophisticated understanding of “neurophysiology and neuroanatomy” and had “multiple correct answers.” Some test-takers were presented with a picture of the cadet chapel of the Air Force Academy in Colorado Springs and asked about its aesthetic purpose. Landy visited the Academy’s Web site looking for an answer, and found that none of the multiple-choice options listed on the test matched the explanation the Academy gave. …

In a conversation with me, Joshua Sohn, the lead attorney for the plaintiffs, recalled that on the last, teachers were asked to explain the meaning of Andy Warhol’s famous soup-can paintings. He questioned why a kindergarten special-education teacher or a Spanish teacher needs to know this; in his view, it’s not a meaningful measure of how well they can do their jobs. But the question also gets to the heart of the case: “The further away the test gets from evaluating basic skills—reading, writing, math—the more likely it is to have some type of difference performance by group,” Sohn told me. “It necessarily tests culture.”

The plaintiffs’ ultimate argument is not that white people know about Andy Warhol and Black people don’t. It’s that asking experienced teachers to correctly interpret the meaning of soup cans is not actually job-related—it’s closer to a poll test than a carefully considered question of competency.

But being a teacher is a pretty generalist jobs, so it makes sense to test broadly for cognitive skill and general knowledge.

Way back in the 1966 Coleman Report that was funded by the famous 1964 Civil Rights Act, sociologist James S. Coleman found that most inputs didn’t measurably affect the output of student test scores. But one that did was how teachers did on a vocabulary test. That sounds plausible both for the direct reason that teachers use words to teach so the more accurately you understand words the more accurately you can teach. Plus, vocabulary is a standard component of many IQ tests, and it’s probably the single best readily-at-hand test of IQ.

Unfortunately, black male teachers scored badly on the test and their students did badly on average. But with the end of Jim Crow segregated schools in sight, Coleman didn’t want to get a lot of black male teachers fired just for being bad at teaching. So he covered that finding up. In his autobiography he said he regretted that decision.

They do the opposite on fire fighter hiring and promotion exams and make all the questions ultra germane to modern fire-fighting techniques. Not surprisingly, this leads to complaints that guys who grew up talking firefighting around the table with their fireman dads and uncles have a big advantage.