Evaluating Observer Ratings: The Case of Measuring Neighborhood Disorder.

The purpose of this study is to evaluate the quality of observer ratings of neighborhood disorder using a many-facet Rasch model (MFRM). Our goal is to investigate observer severity and observer consistency. Observers trained in the use of a systematic social observation visited and rated residential neighborhoods. Data for this study are drawn from the Family and Community Health Study (FACHS). The FACHS sample consisted of 673 neighborhoods. Two observers, out of a total of 67 observers used for this study, rated each residential neighborhood. The results of this study suggested that there were statistically significant differences in observer severity, even after observer training, and that the ratings of observers are not consistent. Therefore, more or better observer training is necessary. In addition, the interaction effect between observer and item was significant, indicating significant variance in observer severity across at least one item.