Common Core: Gary Stern on Manipulating Test Outcomes Through Cut Scores

Maria Baldassarre Hopkins hit the nail on the head: "There are ways to make the numbers do what you want them to do." There was nothing democratic about the way the Common Core was written and forced on the schools. This story will get uglier and uglier as corporate politicos use their manipulated message of failing public schools as a diversion from the real issue of poverty. Reminder: Here's how the New York Times editorial board gushed over the first round of New York's Common Core testing... Ohanian Comment at newspaper site

I disagree with panelists who they believe that once teachers are trained, scores will substantially rise. They will not... Carol Burris

"A small shift in the cut scores means a dramatic difference in the number of students at different levels," said David Dickerson, an associate professor of mathematics at SUNY Cortland who took part. .... Teachers and college professors on the cut-score panel were paid $175 per day, and all panelists were reimbursed for expenses. — Gary Stern, The Journal News

Susan O has been pumping stuff out the past few days and I'm finding it hard to keep up. Take it easy on us old guys Susan. And you're taking away my beach time.

You don't see Mulgrew wanting to punch people in the face over stuff like this. I do and I'm not ashamed to admit it.

http://susanohanian.org/core.php?id=733

(Also see another Susan piece I posted at Norms Notes on New Leaders, New Schools Scams - Another Ed Deform Crook).

Common Core: Who's on track for college and who is not?

Carol Corbett Burris, Principal at South Side High School, Comment:
Gary, thank you for doing this reporting. You are remarkable for your willingness to follow up on stories that are important and difficult to cover. The college readiness information that was shared with the group came from a NYSED sponsored study that said SAT scores which when combined equal a 1630 indicate "college readiness". Nationally, 32% of all seniors get that score. It is no surprise that 33% were proficient on the tests. That standard is needlessly high.

I disagree with panelists who they believe that once teachers are trained, scores will substantially rise. They will not.

Teacher Comment:
The most important statement in this article is that the tests by Pearson were incredibly bad. They were, and continue to be, terrible. My third graders took a reading test where it has been confirmed the stories they had to read and respond to were on a fifth-sixth grade reading level. How does that do anything but frustrate children? These tests are invalid in my opinion and that of most educators. They cannot give useful information about what our children can and cannot do. We need to speak up and demand fair testing. Imagine if as an adult you prepare for the LSAT to take the bar exam (which is shorter in duration than the 3rd grade tests BTW) and when you show up to take the test you have been given the MCAT instead and when you don't do well you are labeled as lacking proficiency in the subject matter. That is what is being done to our children. We must end this pointless, endless testing.

"There are ways to make the numbers do what you want them to do."--Maria Baldassarre Hopkins, assistant professor of education at Nazareth College in Rochester

by Gary Stern, gstern@lohud.com

Within weeks, the state Education Department will release results from the second round of new math and English tests, and tens of thousands of parents will again try to decipher the state's 1-4 scoring system.

How does the state determine the crucial break between a 2, which means that a student is not quite proficient in, say, fifth-grade math, and a 3, which signifies that he or she is on track for college?

These scoring scales were set last summer by a group of 95 educators that the state gathered at a hotel in Troy for several days. Teachers, administrators and college professors from across New York signed confidentiality agreements and were given the task of setting the cuts between 1 and 2, 2 and 3, and 3 and 4 for the new tests. But the scores would be widely questioned and even ridiculed after one-third of New York students were deemed to be on pace.



"A small shift in the cut scores means a dramatic difference in the number of students at different levels," said David Dickerson, an associate professor of mathematics at SUNY Cortland who took part. "It was a contentious process. I think we came up with something that made us all equally unhappy but that we could live with."

The Journal News obtained the list of panelists from the Education Department after months of wrangling and sought to contact all 95 who served. In recent weeks, nine panelists agreed to telephone interviews, five emailed statements or answers to questions and four agreed to describe the process but asked not to be identified.

Panelists who responded were acutely aware of the statewide backlash against the new Common Core-based tests for grades 3 to 8 and the scoring system. Legislators took steps to temporarily reduce the impact of test scores on student placements and teacher evaluations, but the second round of results will still be closely analyzed by schools and parents.

Some panelists defended the scoring system and some reluctantly accepted the results, while others came away feeling the process was so tightly controlled that the results were inevitable. Educators also shared wide-ranging concerns about the Pearson-written tests themselves, the heightened role of testing today, and a lack of equitable teacher training on the Common Core.

"I think the cut scores are fair in an ideal situation, but the kids don't have the background in the Common Core and many teachers haven't been trained," said Stacey Caruso-Sharpe, a math teacher in Amsterdam, New York, who took part in developing the Common Core.

Making the cut

To most parents, passing a test means earning 65 out of 100 points. Cut and dried.

The process of setting a scoring "scale" and cut scores for an annual test, based on all-important, predetermined goals, is an entirely different animal that is not easily described. In fact, the panelists met to set the 1-4 cut scores after students took the first new tests in spring 2013 and the raw data was in.

"It's like you're jumping over a hurdle that's 2 feet high, but after you jump they say it was 3 feet and you missed," said Cary Grimm, another panelist who is math chairman for the Longwood school district on Long Island.

In brief, panelists were assigned to small groups that looked at several grades' exams in math or English language arts. They were given detailed descriptions of what students should know in each grade — prepared by state officials and experts from Pearson Inc., the mega-corporation signed to create New York's tests.

Panelists were also handed all the questions on a given test, from easiest to hardest, according to the experts. Then each educator had to put tabs where he or she thought the cuts should be between 2 and 3 (the most important break), then 1 and 2, then 3 and 4.

Several rounds of discussions and revisions followed. At various points, state officials offered key bundles of data, including actual student results, that helped shape the conversations. Panelists were told whether various cut scores would jibe with research on what it supposedly takes to succeed in college.

Jane Arnold, an English professor at SUNY Adirondack, said the Pearson people provided confusing data that didn't seem to apply to grades 3-5, her group's focus.

"Then they gave us a chance to change our minds," she wrote in a statement. "In other words, we all knew that most of the student scores would be substandard."

Were the cuts fair?

Robert Curry, dean of the School of Arts and Sciences at Alfred (N.Y.) State College, said the process reflected the views of educators from many backgrounds.

"I'm comfortable with the results," he said. "I wasn't happy that so many students got 2s, but neither was I surprised. I see these students come to college, and I know they are not prepared."

But Maria Baldassarre Hopkins, assistant professor of education at Nazareth College in Rochester, said the process was driven by the introduction of outside research about student success.

"I question how much flexibility and freedom the committee really had," she said. "The process was based solely on empirical data, on numbers. ... There are ways to make the numbers do what you want them to do."

Tina Good, coordinator of the Writing Center at Suffolk County Community College, said her group produced the best possible cut scores for ELA tests in grades 3 to 6 — playing by the rules they were given.

"We worked within the paradigm Pearson gave us," she said. "It's not like we could go, 'This is what we think third-graders should know,' or, 'This will completely stress out our third-graders.' Many of us had concerns about the pedagogy behind all of this, but we did reach a consensus about the cut scores."

Eva Demyen, superintendent of the Deer Park district on Long Island, said she still doesn't grasp how the state determined that two-thirds of students were not proficient in English and math.

"How they got the 33 percent (passing) was beyond us," she wrote. "It just seemed very strange to me ... and I'm a mathematician!"

The use of scoring scales for standardized tests is nothing new. More than two dozen states have used various forms.

New Yorkers have questioned the setting of cut scores before. In 2009, when math results boomed, some accused then-Commissioner Richard Mills of lowering the scoring bar. This year, some scoffed when the state said cut scores for tougher high school Regents exams would be set so failure rates would not increase.

A 2006 primer on cut scores prepared by the Educational Testing Service found that cut scores can be reliable, but are based on a group's opinions.

"It is impossible to prove that a cut score is correct," the report said.

Other concerns

Even panelists who thought the cut scores were fairly calculated had problems with the larger goals of New York's testing program.

Derrick Hopkins, a teacher for the small Willsboro school district in northern New York, said anger over test results has distracted the public from positive changes inspired by the Common Core.

"The state is trying to help kids get ready to compete in a global job market," he said. "I don't feel like they are trying to set kids up to fail. We should be looking at the standards, but we started testing before we could teach the Common Core."

Another panelist, Karen DeMoss, a professor of education at Wagner College on Staten Island, said she is increasingly convinced that standardized testing is "scarring" students and not promoting achievement.

"Our process was perfectly fine, and the Common Core standards may be the best thing the country has ever had in education," DeMoss said. "The problem is the underlying assumption that these tests are helping us. They're not. Pearson's tests were unbelievably bad, the worst I've seen, and the reality of using tests designed to rank students is something we haven't gotten our heads around."

Beverly Voos, a retired math teacher from Pittsford, New York, who now trains teachers on the Common Core, said the state has undermined its work by failing to provide equitable teacher training. The state's two main vehicles — running conferences in Albany and funding online lesson plans — have not reached enough people, she said.

"Of course test scores were lousy," Voos said. "Many students have been educated by teachers without training in the Common Core. ... The Common Core has created an ever greater divide between the haves who can afford training and the have-nots."

Twitter: @garysternNY

Compensation

Teachers and college professors on the cut-score panel were paid $175 per day, and all panelists were reimbursed for expenses.

— Gary Stern
The Journal News
July 26, 2014
http://www.lohud.com/story/news/education/2014/07/26/common-core-cut-scores-examined/13219981/