Name
Abstract | ||
|---|---|---|
A global optimization method is introduced that minimize the rate of misclassification. We first derive the theoretical basis for the method, on which we base the development of a novel design algorithm and demonstrate its effectiveness and superior performance in the design of practical classifiers for some of the most popular structures currently in use. The method, grounded in ideas from statistical physics and information theory, extends the deterministic annealing approach for optimization, both to incorporate structural constraints on data assignments to classes and to minimize the probability of error as the cost objective. During the design, data are assigned to classes in probability so as to minimize the expected classification error given a specified level of randomness, as measured by Shannon's entropy. The constrained optimization is equivalent to a free-energy minimization, motivating a deterministic annealing approach in which the entropy and expected misclassification cost are reduced with the temperature while enforcing the classifier's structure. In the limit, a hard classifier is obtained. This approach is applicable to a variety of classifier structures, including the widely used prototype-based, radial basis function, and multilayer perceptron classifiers. The method is compared with learning vector quantization, back propagation (BP), several radial basis function design techniques, as well as with paradigms for more directly optimizing all these structures to minimize probability of error. The annealing method achieves significant performance gains over other design methods on a number of benchmark examples from the literature, while often retaining design complexity comparable with or only moderately greater than that of strict descent methods. Substantial gains, both inside and outside the training set, are achieved for complicated examples involving high-dimensional data and large class overlap |
Year | DOI | Venue |
|---|---|---|
1996 | 10.1109/78.553484 | IEEE Transactions on Signal Processing |
Keywords | Field | DocType |
design complexity,strict descent method,novel design algorithm,global optimization technique,statistical classifier design,global optimization method,data assignment,radial basis function design,annealing method,classifier structure,design method,deterministic annealing approach,statistical physics,physics,annealing,constrained optimization,training set,learning vector quantization,radial basis function,constraint optimization,probability,multilayer perceptron,information theory,back propagation,backpropagation,algorithm design and analysis,error probability,high dimensional data,cost function,design optimization,free energy,probability of error,statistical analysis,global optimization,entropy | Information theory,Mathematical optimization,Global optimization,Computer science,Learning vector quantization,Multilayer perceptron,Classifier (linguistics),Backpropagation,Randomness,Constrained optimization | Journal |
Volume | Issue | ISSN |
44 | 12 | 1053-587X |
Citations | PageRank | References |
44 | 4.54 | 33 |
Authors | ||
4 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| David J. Miller | 1 | 421 | 60.29 |
| Ajit V. Rao | 2 | 197 | 20.50 |
| K. Rose | 3 | 44 | 4.54 |
| Allen Gersho | 4 | 2031 | 624.48 |