Testing Expert Fred Smith Calls for Boycott of Flawed Tests

Leonie posted this post with fabulous links and I'm cross-posting due to the importance of Fred Smith's work.

The GEM high stakes testing committee, which Fred has worked with, has been working on supporting parents who are willing to do a test case in NY State, one of the only states without a policy) by opting out of the test (do their kids get left back?). Even within the anti-hst community there is some controversy as to whether pushing a program of opting out is worthwhile at this point (if high scoring kids opt out the school rating goes down) but there will be a push forward.


From Leonie Haimson: Feb 22, 2012
http://nycpublicschoolparents.blogspot.com/2012/02/testing-expert-points-out-severe-flaws.html

Testing expert points out severe flaws in NYS exams and urges parents to boycott them this spring!

Many recent columns have pointed out the fundamental flaws in the new NY teacher evaluation system  in the last few days: by Aaron Pallas of Columbia University; Carol Burris, Long Island principal, education historian Diane Ravitch, (who has written not one but two excellent critiques) and Juan Gonzalez, investigative reporter for the Daily News.  All point out that despite the claim that the new evaluation system is supposed to be based only 20-40%  on state exams, test scores in fact will trump all, since any teacher rated "ineffective" on their students' standardized exams will be rated "ineffective" overall.

To add insult to injury, the NYC Department of Education is expected to release the teacher data reports to the media tomorrow -- with the names of individual teachers attached.  These reports are based SOLELY on the change in student test scores of individual teachers, filtered through a complicated formula that is supposed to control for factors out of their control, which is essentially impossible to do. Moreover, there are huge margins of error that mean a teacher with a high rating one year is often rated extremely low the next. Sign our petition now, if you haven't yet, urging the papers not to publish these reports; and read the outraged comments of parents, teachers, principals and researchers, pointing out how unreliable these reports are as an indication of teacher quality.   

Though most of the critiques so far focus on the inherently volatile nature and large margins of error in any such calculation, here in NY State we have a special problem: the state tests themselves have been fatally flawed for many years.  There has been rampant test score inflation over the past decade; many of the test questions themselves are amazingly dumb and ambiguous; and there are other severe problems with the scaling and the design of these exams that only testing experts fully understand.  Though the State Education Department claims to have now solved these problems, few actually believe this to be the case.

As further evidence, see Fred Smith's analysis below.  Fred is a  retired assessment expert for the NYC Board of Education, who has written widely on the fundamental flaws in the state tests.  Here, he shows how deep problems remain in their design and execution -- making their results, and the new teacher evaluation system and  teacher data reports based upon them, essentially worthless.  He goes on to urge parents to boycott the state exams this spring.  Please leave a comment about whether you would consider keeping your child out of school for this purpose!


Fred Smith:
New York State’s Testing Program (NYSTP) has relied on a series of deeply flawed exams given to 1.2 million students a year.  This conclusion is supported by comparing English Language Arts (ELA) and Math data from 2006 to 2011 with National Assessment of Educational Progress (NAEP) data, but not in the usual way.

Rather than ponder discrepancies in performance and growth on the national versus state exams, I analyzed the items—the building blocks of the NAEP and NYSTP that underlie their reported results.

 

NAEP samples fourth and eighth grade students in reading and math every two years.  Achievement and improvement trends are studied nationwide and broken down state-by-state.  The results spur biennial debate over two questions:

Why are results obtained on state-imposed exams far higher than proficiency as measured by NAEP?  Why do scores from state tests increase so much, while NAEP shows meager changes over time?  

 

By concentrating on results—counting how many students have passed state standards; or trying to gauge the achievement gap; or devising complicated value-added teacher evaluation models, school grading systems and other multi-variate formulas—attention has been diverted from the instruments themselves.

In 2009, the year statewide scores peaked, Regents Chancellor Merryl Tisch discredited NYSTP’s implausibly high achievement levels and low standards.  She pledged reform—“more rigorous testing” became the catch phrase.  She knew that New York’s yearly pursuit and celebratory announcements of escalating numbers belied a state of educational decline.

 

It was an admission of how insidious the results had been, not to mention the precarious judgments that rested on them. Unsustainable outcomes became a crisis.  But Albany refuses to address the core problem. 

 

The NAEP and NYSTP exams contain multiple-choice and open-ended items.  The latter tap a higher order of knowledge.  They ask students to interpret reading material and provide a written response or to work out math problems and show how they solved them, not just select or guess a right answer.

 

Ergo, on well-developed tests students will likely get a higher percentage of correct answers on machine-scored multiple-choice questions.  In addition, the same students should do well, average or poorly on both types of items.   

 

NAEP meets the dual expectations of order and correlation between its two sets of items; the NYSTP exams, which are given in grades three through eight, do not.  Here are the contrasting pictures for the fourth grade math test.  

 

NAEP’s multiple-choice items yield averages that are substantially higher than its open-ended ones.  The distance between them is consistent over time—another way of saying that the averages run along parallel lines.  There is an obvious smoothness to the data.

Items on the NYSTP exams defy such rhyme and reason.  The percent correct on each set of items goes up one year and down the next in a choppy manner.  In 2008, performance on the teacher-scored open-ended items exceeded the level reached on the multiple-choice items. In 2011, when the tests were supposed to have gained rigor, the open-ended math questions were 26.2% easier than the NAEP’s.

                       

The reversals reveal exams made of items working at cross purposes, generating data that go north and south at the same time.  That’s the kind of compass the test publisher, CTB/McGraw-Hill, has sold to the State since 2006 to the tune of $48 million.

 

I found incongruities in all grades measured by NYSTP.  On the ELA, divergent outcomes on the two types of items are noteworthy for grades 5, 6 and 7 in 2010 and 2011.  The fifth grade items provide a jarring illustration, because they continue to function incoherently in the years of promised reform. Averages on the open-ended items increase (by 10.7%), as sharply as the multiple-choice averages fall (10.3%)—crossing over them last year.

It all goes unnoticed.  Press releases are written in terms of overall results without acknowledging or treating NYSTP’s separate parts.  This is odd since so much time and money go into administering and scoring the more challenging, higher-level open-ended items.

 

So, we’ve had a program that has made a mockery out of accountability, with the head of the Regents running interference for it.  Parents watch helplessly as their children’s schools become testing centers. And the quality of teachers is weighed on scales that are out of balance, as Governor Cuomo takes a bow for leveraging an evaluation system that depends on state test results to determine if a teacher is effective.  

 

If the test numbers aren’t good enough, there’s no way teachers can compensate by demonstrating other strengths needed to foster learning and growth.  Within days it is likely that newspapers will publish the names of teachers and the grades they’ve received based on their students’ test scores going back three years.  As shown, however, these results are derived from tests that fail to make sense.

 

The graphs are prima facie evidence that the vendor and Albany have delivered a defective product. High-stakes decisions about students, teachers and schools have depended on it. An independent investigation of NYSTP is imperative to determine what happened and, if warranted, to seek recovery of damages.  

 

It is also time for the victims—parents in defense of their children, and teachers in support of students, parents and their own self-interest—to band together and just say “No!” to this April’s six days of testing.

---Fred Smith, a retired Board of Education senior analyst, worked for the city public school system in test research and development.