The Theory of Group Operating Characteristic Analysis in Discrimination Tasks

Abstract

Inconsistent decision making is a long-standing problem in psychophysics, where decisions based on the same stimulus often differ across replications of an experiment. Inconsistency is described statistically by the concept of unique noise, the effects of which are removed by averaging ratings across replications on a per-stimulus basis. A group operating characteristic (GOC) curve is a type of receiver operating characteristic (ROC) curve based on the mean rating per stimulus. GOC analysis is shown to improve task performance dramatically compared to ROC analysis, and can recover theoretical ROC curves from noisy data. This thesis presents a theory of GOC analysis showing why the procedure works. It also develops transform-average GOC analysis, transfer function analysis, and shows how to estimate unique-noise-free performance from a finite, unique-noise-affected data set. Transform-averaging of ratings (for example, by using geometric or harmonic means) extends GOC analysis to include strictly monotonic increasing (s.m.i.) transformations of rating scale data. Although s.m.i. transforms do not alter ROC curves on any single replication, it is shown that they do alter GOC curves because of unique noise. Nevertheless, GOC analysis may be transform-invariant, apart from residual unique noise effects. Empirical evidence is given showing how GOC performance improves towards theoretical performance regardless of the particular rating scale that is involved. A psychophysical transfer function is an s.m.i. mapping from a decision axis onto a rating scale. Transfer functions underlie theoretical interpretation of empirical ROC analysis, and it is shown how they can be estimated from empirical data. The theory of GOC analysis incorporates both transfer functions and transform-average GOC analysis under the same framework. The theory shows that GOC analysis will work under arbitrary (and possibly unknown) transfer functions, and under arbitrary ordinal scalings of a rating scale, but only when a family of unique-noise-affected evidence distributions are stochastically ordered on the decision axis. If stochastic ordering does not hold, unique-noise-free GOC performance changes according to the scaling of a rating scale. When that is the case, empirical results and subsequent theoretical interpretation become somewhat arbitrary. This finding about unique-noise-affected rating scales also extends to theoretical models that incorporate unique noise. Without stochastic ordering on a decision axis, the theoretical unique-noise-free ROC curve can change following an s.m.i. transform of the decision axis. GOC performance improves as a function of replications added (FORA). Stable empirical FORAs result from all combinations analysis (ACA), where average performance is calculated over all possible GOC curves for a given number of replications. The logarithm of FORA increments is generally a linear function of the logarithm of the number of replications, typically with r2 > 0.995. This pattern implies a three-parameter data model that provided an excellent description of FORAs from six different experimental projects. These projects involved different aural discrimination tasks, experimental paradigms, decision methodologies, individual observers, levels of performance, stimulus parameters, and measures of sensitivity. Dozens of different FORAs followed the same mathematical form—only the three parameters of the data model changed. Extrapolation of a FORA to an infinite number of replications makes it possible to estimate asymptotic unique-noise-free performance and its sample statistics based on a finite data set. Empirical FORA analysis showed that the observer with the best (unique-noise-affected) ROC performance was often not the observer with the best unique-noise-free performance. This shows that unique noise can generate deceptive results in psychophysics, but that its effects can be removed by using GOC analysis.