A comparison of linear discriminant function and vertical percent methods for weighting application form responses: a Monte Carlo simulation

Abstract

This study compared the predictive accuracy of two empirical weighting methods, the Vertical Percent Method (VPM) and the Linear Discriminant Function (LDF) approach, for scoring application form item responses. A Monte Carlo procedure was used to randomly generate a large number of item response alternatives for a 20-item pseudo-application form over a series of 500 trials. Using development and cross-validation sample the two weighting methods were compared primarily for predictive accuracy. The chi-square analysis was used to compare samples within methods and the McNemar test to compare methods within samples. The results indicated that the difference in the percentage correctly classified into criterion groups between development and cross-validation sample was significant for both weighting methods. Although both methods displayed significant shrinkage in classification accuracy on cross-validation, the LDF procedure produced fewer trials displaying significant differences. The classification accuracy in the majority of trials in the development sample favoured the Vertical Percent Method, however neither method displayed unequivocal classification superiority in the cross-validation sample. These findings and the practical considerations of using these methods to weight application form information are discussed with recommendations for the use of these empirical weighting strategies.