Fitting a model whose explicit formula is unknown, with application to ordered contingency tables

Abstract

There are many contexts in which exact measurement is impossible or impracticable, and human judgment (more or less expert) is resorted to. Studies of the reproducibility of such judgments are common, and often include two-way tables of frequencies with both variables ordered --- for example, proficiency in speaking Russian as judged by two raters, or health of plants as judged by two raters. It may happen that the bivariate normal distribution is a good fit to the data. But suppose this is not the case, and instead there is asymmetry in the table of frequencies, with the raters disagreeing more about the relatively expert Russian speakers than about the novices, or about the relatively healthy plants than about the less healthy? A bivariate distribution is invented, the chief features of which are that it is a variables-in-common model, true score plus error for each rater, and that the scatter of error is greater when the true score is high than when it is low. (That is, the homoscedastic assumption of the bivariate normal distribution no longer holds.) It seems impossible to obtain an explicit expression for the joint distribution of the two observed ratings, and even if some expression involving special mathematical functions could be written down, it would be hopelessly difficult to use. This impasse is by-passed by fitting the distribution to the data by computer simulation. The software used has ranking and recoding commands of one line each, so it is easy to ensure the fitted marginal distributions exactly match the data, and it is unnecessary to estimate parameters representing the boundaries between the grades of rating. This procedure may be useful with many other datasets of this type, as well as with those discussed here.