This paper examines CPU performance data for 209 systems, as published by Computerworld from 1981 to 1984, and correlates performance with system configuration parameters. The authors found a relationship between processor performance and a combination of three parameters: cache memory size, average number of channels (the arithmetic mean of the minimum and the maximum), and the average main memory size (similarly defined). Beyond a threshold that corresponds roughly to the performance of a VAX-11/730, or 0.3–0.4 MIPS, the authors’ formula yields performance figures that are, on the average, within 30 percent of the published figures. As speed increases, variation drops further.

The ability to predict CPU performance from a few easily obtained numbers is of great potential value. The authors cite three reasons: performance figures such as those in Computerworld cannot be obtained independently by those without access to Computerworld’s methods; figures are not published for all computers; and figures cannot be determined for machines not yet in production.

Unfortunately, the method of this paper does not give us this ability. The method is descriptive, not predictive. It says, in effect, “if a system has the same cache size, number of channels, and main memory range as another, its CPU is probably about as fast,” and it suggests how decreases in one of these areas can be balanced by increases in another to yield the same conclusion. But the number of channels offered with a system (for example) can in no way affect CPU performance. Rather, it is chosen after the fact by product planners--based in large part on what they are told of CPU performance--in an effort to provide sufficient capacity for most uses without unduly burdening low-end systems with excess cabinetry and bus slots. The authors ask us to infer CPU performance by assuming the planners have done their work well and by working back from their choices. If channel count and memory size are of any predictive value, this reflects the assumption that planners will make these choices in the future much as they have made them in the past.

This circular reasoning leads to illogical results. For example, the Honeywell DPS 6/9X CPU was offered in both a small and a large package. The model predicts that a DPS 6/96 should be 21-2- times as fast as a DPS 6/92, while they actually have the same CPU and hence the same CPU performance.

A second problem is inconsistent definitions (by vendors and/or Computerworld) of channel. The Digital VAX-11/780 has up to eight channels, meaning unibus or massbus slots. The Honeywell DPS 6/96 can have 112 channels, here meaning terminals and/or communication lines. The Wang VS 100 has no channels at all. These three systems are of similar performance in practice. This is probably why the authors found that channel quantity is less correlated with CPU performance than is main memory range. Main memory size definitions are, at least, consistent.

A third shortcoming of the authors’ formula is that its inputs are available only after complete systems are placed on the market. When a system is developed in toto by one manufacturer, one can at least hope that its planners will create reasonable relationships between the performance and the configuration. But what is one to do with a microprocessor? What is the performance of a 20 MHz Motorola 68020? is a meaningful question. But a chip may lack cache, while cache can be included in a system. A chip has no main memory, though its architecture may impose an upper limit. A chip has no channels. The method of this paper applies only to processors that have been incorporated into systems. Since the paper purports to address CPU performance, this is a serious difficulty. It is made more serious by the fact that a given microprocessor can be the basis of a wide variety of systems having different market targets, different overall throughput, and hence, quite properly, different configuration limits.

This leads to a fourth problem. The authors appear confused between CPU performance and system throughput. They refer (p. 312) to “the well-known fact that . . . interactions among main memory size . . . and number of channels affect CPU performance.” This “well-known fact” is simply not true. The cited factors affect system throughput but cannot affect CPU performance.

Statistically, this paper demonstrates a self-fulfilling prophecy. The authors begin with parameters that have a behavioral (but often not technical) connection to CPU performance and select three that correlate best with the independent variable. It is not surprising that they find low variation from the values thus predicted. The proper approach to such problems is to determine the prediction formula from one randomly selected subset of the data points and to validate it with the remaining data. The authors omit this elementary step, leaving their results highly suspect on statistical grounds.

This paper is of value for collecting four years’ worth of published performance data in one place. It is of interest for finding average historical relationships between a few system configuration parameters and processor performance. As a means of predicting processor performance, its methods are of little use. Manufacturers’ claims, MIPS ratings (for all their well-documented shortcomings), and competitive positioning are all easier to come by and just as useful.