It’s Not About the Score; It’s the Cut-Score

***As this information continues to change and develop, this post will be updated.

There has been a lot of news about cut-scores in the last few days; but the only thing people really seem to understand, is that they don’t really understand cut scores. So, let’s break it down.

What is a “cut score”?
This one is pretty straight-forward:  It is a cut-off point. If you had a number line, and divided it by what scores are advanced, which are proficient, etcetera, those dividing points are the “cuts.”

What is a “quick score”?
A quick score is a score is temporary. It is delivered quickly, after the tests have been administered, to give students and teachers a score that can be used for final grades and or placement in the next year’s courses.

If these tests are as carefully created and reliable as they claim to be, why can’t they deliver the real score, rather than a “quick score”?
That’s a good question. Keep asking it. (I’m pretty sure you already know the answer.) If anyone gives you an answer that doesn’t involve manipulating test data, question that, too.

What is an “equated” score?
This is just taking a score and making it comparable to another score. For example, we might have one test that produces a score of 1 to 5, and a similar test that produces a score of 0% to 100%. Comparing those, a score of 4 would mean two very different things. To make a comparison, we might “equate” (translate) the 4 to 80%.

That sounds easier than it is. When creating an “equated” score, additional factors are taken into consideration, beyond just the number. One of those tests may have questions that are far more difficult than the other, for example. So, to make an equated score, the difficulty of each test is also considered, along with the number of questions, the kinds of questions, and other factors, to get down to what a score on one of these tests would “equate” to, on the other.

What are “pre-equated” and “post-equated” scores?
“Pre-equated” scores are scores that are equated BEFORE students are tested.

“Post-equated” scores are equated AFTER students are tested.

Why does it matter whether cut scores are equated before or after testing?
Now we are in a tricky area. It looks simple; but it is not.

(If you would like to read a paper on the topic, A Comparison of Pre-Equating and Post-Equating using Large-Scale Assessment Data explains things well.)

Most organizations use PRE-equated scores, because they are better able to justify where the cut-offs occur. If an test creator has carefully considered their questions, the course content those questions map to, and the difficulty of the questions, they should be able to reasonably set the cut-off points where students should be expected to perform if they have “basic” knowledge, “proficient” knowledge, or “advanced” knowledge.

Since the Tennessee Department of Education uses Post-equated scores – they can willy-nilly set the cut scores to whatever they want, AFTER the tests have been taken.

This changes EVERYTHING your students and teachers have been told. They were given curriculum to use and scores to work toward. Their “quick scores” showed that they mastered the material. But the Department of Education gets to go back and CHANGE THEIR MINDS about the cut-scores.

Can you imagine being in a class, where the teacher gives you a grade and over the summer, sends you a note that he re-calculated the grades and your grade is completely different?!  That is what the Tennessee Department of Education can do by determining cut-scores AFTER the tests have been taken and scored.

A local reporter told me that she is trying to do a story on this, but is having trouble getting information from the Department of Education!  How is that acceptable? Why aren’t parents on the doorstep of the TN DOE?

 

The Tennessee Education Association has also been trying to get answers and posted this information on May 27, 2015, on their Facebook page:

TCAP Update:
Following the state’s conference call, we now know that the state did change its methodology for calculating quick scores for students in grades 3-8. It is now using the cubed-root method the state has been using for high school EOCs. This change in methodology resulted in apparent grade inflation, leading parents and educators to believe students had performed better than in previous years. The change resulted in about a 4-point increase in cut scores from the method used in 2014.

Please visit the link below for documents provided by the state in its attempt to explain these changes. TEA still has many, many questions about the reliability of both the quick and cut scores, why these changes were made and how proficiency levels are determined. We will continue our efforts to get more answers from the state and insist that they‪#‎showthemath.

State Documentation of TCAP scores

Below are some of the official answers TEA has been able to get, so far. Please note that TEA has been doing their due diligence on this issue and there has been more information, each time I have looked at their page.  Please use the link above, to follow their findings.

Quick Score/Proficiency level correlation:
We have not changed the mark or expectation for student proficiency on TCAP; there have been no changes to cut scores for proficiency levels. I’d also like to clarify that quick scores are no longer tied to TCAP performance levels. For example, a quick score of 85 is not equivalent to the cut score for proficient. We compare student performance each year based on the scale scores.  The scale scores determine the cut points for performance levels (i.e. below basic, basic, proficient, advanced). We always produce equating tables in the fall that clearly define the raw score equivalent cut points based on the scale score. This is designed to help teachers know what to expect early in the school year. The equating tables for 3-8 achievement can be found here.  The equating tables for EOC can be found here.

Student performance expectations for the proficiency threshold have not changed.  They are exactly the same as last year, and these expectations are exactly the same as the equating tables which we published online in the fall for teachers to access. Quick scores do not determine proficiency levels. I have attached a FAQ – A Guide to Understanding Quick Scores – that we created to help explain the purpose for quick scores.  In addition, please see the attached TCAP Scoring Flow Chartthat shows how and where quick scores fall into the scoring process.  It is clear from the flow chart that quick scores have no relationship to performance levels.  Quick scores are used only to calculate a 100-point grading scale. There are various methodologies that can be used to create a 100-point grading scale from the raw score, and, this year, we used the cube root method for grades 3-8, as we have done for EOCs over the past several years.

Quick Score Calculation:
What was the rationale for making this change to the cube root method? Is it possible to see the formula used for this calculation?

The rationale for making the change was to create a consistent methodology for generating quick scores and one that was not dependent upon TCAP performance levels like the interval scaling method used in 3-8 achievement since 2012. We updated the methodology to be consistent with what we are doing for End of Course exams.  We will be engaging directors of schools in more conversations about quick scores for 2015-16.

I have attached (linked above) a memo from April 2012, TCAP Quick Score Conversion Guidance, which includes the interval scaling methodology for generating quick scores in grades 3-8.  I have also attached the Cube Root Quick Score Calculation guidance that details the cube root method used this year for all grades.

Proficiency Levels:
What are the proficiency level ranges for Below Basic, Basic, Proficient, and Advanced for the various assessments? How do these ranges compare to previous years?

The equating tables for 3-8 achievement and EOCs are posted online, and they show the scale score ranges for each performance level.  These scale score ranges are the same for 2015 as they were in 2014.

 

The fact that teachers, districts, parents, and communities are having difficulty getting timely and adequate answers from the State Department of Education should be very concerning. It certainly makes things look fishy.

Next Up: High School EOC Cut Scores, Predicted Scores, and Misuse in Teacher Data

Problems Didn't Start Yesterday

A supporter contacted me to ask that I make some of the dates in this article more clear.  While doing so, I have also added pieces of the video transcripts and made my photos from the 2011 Rally more accessible.

One thing that I cannot add, is YOUR voice – and I hope you will help, by adding  your experiences in the comments.  If you have spent years, trying to communicate these problems to our Board and/or Legislators, please add your voice.  (No full names will be published – Make one up, if you like.)  They like to claim that there is only a “small group” of 22 “disgruntled” teachers, who are trying to stir things up.  What they need to know, is that there are THOUSANDS of teachers, parents, students, and community members, who are begging to be heard.

——————————-

The problems in Knox County Schools didn’t start yesterday, and they aren’t going to end tomorrow. Lots of people want you to only notice the past few months. Legislators, school boards, and district administrators want to make out teachers as spur-of-the-moment rabble-rousers, when the truth is very different. They want to convince the public to give them “time” to make changes, hoping the public will forget that we have been explaining the problems for YEARS – not months.

The Knox County Board of Education has heard from THREE different Knox County Education Association presidents. (Elected by teachers, they serve two-year terms.)  They have been hearing about problems for MORE THAN FOUR YEARS.

The Board of Education, as well as local legislators were sent numerous letters and emails from educators – and were repeatedly told that we needed to “wait and see.”

Legislators very purposefully passed these laws. Boards of Education, at the local and state level, supported them. And superintendents “testified” in front of the legislature, claiming that teachers “embraced” the testing and evaluation systems that were being put into place and asked legislators not to change a thing:

Certainly, there are some adjustments and tweaks that can be made,” said McIntyre, adding that those can be done without legislative action. “In my humble and respectful opinion, I would ask that the Legislature keep the legislation in place in its current form.

From:  “Jim McIntyre Defends New Teacher Evaluation System” by Tom Humphrey, Knoxville News Sentinel, November 2, 2011

 

And, two years later, in 2013, McIntyre continued to testify that the new evaluation system was fabulous:

But perhaps no other recent change has greater potential to improve the quality of education in our state than the adoption of a new teacher performance evaluation system.

From: U.S. House of Representatives Early Childhood, Elementary, and Secondary Education Subcommittee, Written Testimony as prepared by Dr. Jim McIntyre, Superintendent 2/28/2013

At this same time, teachers were speaking to the Knox County Schools Board of Education.  According to the BOE, they just “didn’t know” that morale was at an all-time low, or that students were undergoing so many tests, or that the evaluation system was seriously flawed and being used punitively, rather than for improvement.  In other words, they lied.

The KCS BOE has been told about these problems since 2010.  KCEA President, Sherry Morgan, spoke every time they met during the 2011-2013 school years. All three KCEA presidents regularly contacted Board members and legislators between 2010 to today – as have numerous other teachers, parents, and community members.

Four years.

Four years of ignored data.

Four years of ignored facts.

Four years of ignoring the professional educators of Knox County and Tennessee.

Four years of supporting the superintendent’s false testimony.

And, after four years of completely ignoring teachers, parents, and students, they ask us to please, just give it some time to “work” before they make changes.

The only people they are kidding are themselves and people who have been asleep for the past four years.

If anyone believes that teachers were just too softspoken to get through to the BOE, or that the BOE just didn’t “get” that there was a problem, here are some reminders in pictures and video from March 5th, 2011:

Tennessee Education Rally – March 5, 2011

 

Tennessee teachers march to the Capitol, to rally against several anti-public education bills.
March 5, 2011

 

Kayla Montgomery, KCEA, speaking at 2:15
“When people try to get rid of a bill, ask why that bill was in place. In essence, it’s to protect somebody.”
March 5, 2011

 

Though this isn’t a bi-partisan speaker, the thing to remember is that this is no longer about one party or the other.  It is about fixing the MESS that has been created.

0:55  – “This is an attack on teachers; this is an attack on firemen; this is an attack on collective bargaining.  Collective bargaining – there are many entities that do collective bargaining – and you can’t forget what collective bargaining did for this country:  Collective bargaining gave us the weekends; collective bargaining gave us safety laws in the workplace; collective bargaining…not to work children like they were grown people.”

1:45 – “These folks hate public education.  They don’t care anything about public education.  They tried to dicimate it with charter schools; they tried to steal the money everywhere they can, they tried to do away with the DOE at the national level.  They do not care about education.  They never have and they never will.”

2:55 – “This is a plot that’s happening all over the country…They are doing it all over the country.”

 

 

Alexander & Jeb Bush, Supporting CCSS

Where are the educators in this discussion???

“U.S. Sen. Lamar Alexander (R-TN) will appear with former Florida Gov. Jeb Bush (R) and Gov. Bill Haslam (R-TN) Wednesday at an education roundtable discussion sponsored by pro-Common Core Tennessee Chamber of Commerce and Industry and the Tennessee Business Roundtable.”

http://www.breitbart.com/Big-Government/2014/03/18/Tennessee-Sen-Lamar-Alexander-Joins-Jeb-Bush-In-Support-Of-Common-Core-Standards

Arizona Senate Bans Common Core

“Arizona adopted the Common Core standards in 2010 without opposition, but some Republicans have recently denounced the standards, saying they are a poorly conceived, federally driven effort that usurps states’ rights.”

Read more about Arizona’s decision to ban the Common Core State Standards here.

More Money for Pearson & Gates – Since 2011

If you still think “Common Core” standards were developed by anyone in the education field, or that the standards are a new idea, you might want to look at this press release from April 27, 2011:

“NEW YORK–The Pearson Foundation today announced a partnership with the Bill & Melinda Gates Foundation to support America’s teachers by creating a full series of digital instructional resources. Online courses in math and reading/English language arts will offer a coherent and systemic approach to teaching the new Common Core State Standards. Common Core Standards were developed by the National Governors Association, in partnership with the Council of Chief State School Officers. Forty-one states, two territories, and the District of Columbia have adopted the standards.”

Posted in News ReleasesPreK-12