Discrimination Among Events by Neural Networks

Abstract

Usually, the performance of neural networks in event discrimination tasks is measured by the proportion of correct decisions. Although the deficiencies of proportion correct warrant the use of receiver operating characteristic (ROC) analysis, the detectability measures associated with ROC analysis also have deficiencies. Information theory is used to develop a new detectability measure that is free of the deficiencies of the existing detectability measures. The new measure, denoted D2,is based on the area below and the area above the ROC curve. It was used to evaluate how well two neural networks-the Hopfield network and the back-propagation network-distinguish between stored and other patterns. It is shown that the probability distributions of the Liapunov function associated with the Hopfield network can be used to construct ROC curves. For the back-propagation network, ROC curves can be constructed using the posterior uncertainty of the events. The Hopfield network and the back-propagation network were compared, also, to a benchmark nearest-neighbor network. ROC analysis is generalized in order to define appropriate performance indices for neural networks involved in tasks with three or more events. It is shown that the performance of an observer (such as a neural network) in an identification task with n independent events can be represented in n! ROC spaces of dimension n. Each ROC space is associated with a unique pairing of the events and decisions. A hypersurface can be generated in each ROC space by manipulating the observer's decision criteria. It is shown that the hypervolumes of each hypersurface is a probability and that the hypervolumes sum to one. Using these facts, the detectability measure D2 is generalized. The generalized measure, denoted Dn, is nonparametric and independent of the criteria. The value of Dn is shown to increase monotonically with n and to be equal to the channel capacity of an observer in a n-interval forced-choice task. Examples are given of the application of generalized ROC analysis to neural networks. In particular, Dn was used to evaluate how well the back-propagation network can distinguish among sets of up to seven stored patterns. It is concluded that measuring the' performance of neutral networks is a more difficult problem than is generally supposed, and that sound detectability measures such as Dn are required.