Subjectivity in PSSA Scoring

The Technical Report for the PSSA for 2007 is on the PA Department of Education website. This report is compiled by DRC to explain the PSSA test development and implementation process for a given year. After reviewing some items from the Technical Report for the PSSA 2007, I am still upset that the open ended questions have a large degree of subjectivity involved in the grading process. As previously stated in an earlier blog, 10% of the tests are subject to a review by more than one "scorer" to determine if there is accuracy between the grade given by one scorer on an open ended question as opposed to the grade another scorer would give the same question. According to the Technical Report 2007 pg 51, two different scorers have only a 70 to 80 percent chance of giving the same score to the same Reading question. In other words, if two educated and trained professionals read the same answer, the two of them are only 70-80 percent likely to give the same score. Many times the readers will give adjacent scores, (adjacent scores are scores that are beside each other, i.e. a score of 2 versus 3). Of course, that would happen when the answer can only receive a score of zero to 3. (Three is the total points allowable for a Reading open ended question. Math open ended questions can receive scores of zero to four points.) To the state's credit, the Math open ended answers have a higher percentage of scoring accuracy. Perhaps the scores are higher because Math is a more cut and dry subject than is Reading. Then again, perhaps the scorers were just tired the day they reviewed Reading. Maybe even as tired as the child who wrote the answer...