Technical correspondence.

The column, "Benchmarks for LAN Performance Evaluation," by Larry Press (Aug. 1988, pp. 1014-1017) presented a technique for quickly benchmarking the performance of LANs in an office environment. This piqued our interest since office automation is growing in importance. As a result, an empirical analysis of the Press benchmark programs was conducted. The results indicated that these benchmarking programs were appropriate for the benchmarking of LANs in an office environment.

Press presented two resource-oriented programs; one that models intermittent network activity, and one that models constant network activity. These programs were used in a comparison of 18 commercial LAN configurations. For each configuration tested, one workstation was used as the benchmark station. One this station a set of foreground tasks was run and timed, while the other workstations (maximum of 10) simulated background activity.

James E. Smith suggests in his article, "Characterizing Computer Performance with a Single Number" (Oct. 1988, 1202-1206) that the principle of benchmarking is to model a real-world job mix with a smaller set of representative programs. If the benchmark is chosen well, then each job in the real-world job mit has the same performance characteristics as the benchmark program. Consequently, research which empirically evaluated the Press LAN benchmark programs was conducted using office performance data. This analysis compared performance data using the Press benchmark programs to actual performance data. The objective was to determine whether the benchmark progrmas have the same performance characteristics as the real-world job mix.

For an initial analysis, a 4-megabit token-ring network was used running IBM PC LAN version 1.20 with DOS 3.30. The netwrok server was an IBM PC AT 339, with twenty IBM PC XT 286 workstations. The software used was WordPerfect 5.0 network version. The scenario consisted of three tasks--loading WordPerfect 5.0, loading a 50K file from the server within WordPerfect, and then saving a 50K file within WordPerfect to the network server. This experiment was carried out with 1, 5, 10, 15, and 20 active client machines on the network.

Twenty students from a senior-level business computing class were used to conduct experiments which generated actual performance data. A foreground worksation was used for timing with the remaining 4, 9, 14, and 19 workstations in the background performing the same actual tasks as the foreground workstations. The benchmarking performance data was collected by timing one workstation performing the three tasks while the background benchmark programs were run on 4, 9, 14, and then 19 workstations. Each of the benchmark programs used a fixed record length of 80. Different LAN benchmark programs were used to simulate background activity. Performance (timing of tasks) for one workstation using actual task activity in the background was compared to performance for one workstation using the benchmark programs in the background.

Press presents a BASIC LAN benchmark program that opens a file and then processes a loop which writes one record, reads the record and displays the cumulative mean and variance of the times between transactions. This reading and writing of a record is designed to represent constant netowrk activity--CONSTANT-A. CONSTANT-B is a modification of the constant activity program that opens a file, writes a rcord, closes the file, and then executes a loop which opens the file, reads the record, writes the record, and then closes the file. Another modification, CONSTANT-C is developed to more accurately represent constant network activity (input and output) on the network. This program opens a file, writes ten records, and closes the file. A loop is then processed which opens the file, randomly reads a record, randomly writes a record, and then closes a file.

Press also presents an intermittent activity program with the time between transactions distributed normally--NORMAL-A. A modified version of the program, NORMAL-B, opens a file, writes 10 records, and closes the file. The program then loops to process an open file, randomly read a record, randomly write the record, and close the file.

Actual Performance: Before any of the background programs were run, a single workstation took 20.7 seconss to complete the senario--13.30 seconds to load WordPerfect from the server, 2.90 secons to load a 50K file from the server, and 4.50 seconds to save the 50K file on the server. As the number of workstations in the background performing the scenario increased, the time to complete the senario increased at a nonlinear rate. Normalizing the total time to complete the senario to one workstation with 4, 9, 14, and 19 background stations all performing the same tasks as the timing station, it took 4, 9, 15, and 23 times as long, respectively, as a single workstation to complete the scenario.

Benchmark Performance: The three constant network activity and two intermittent activity (normal) benchmark programs were executed on 4, 9, 14, and 19 background stations while on e timing workstation accomplsihed the scenario. CONSTANT-A, since it did not open and close the file, did not compare well to the actual performance for different numbers of background workstations. For all combinations the timing was the same as one workstation performing the scenaio. The CONSTANT-B and CONSTANT-C programs for 4, 9, 14, and 19 background workstations took 3, 6, 9, and 12 times and 4, 7, 11, and 14 times as long as a single workstation to complete the scenario, respectively. The NORMAL-A and NORMAL-B benchmark programs took 4, 8, 12, and 16 times and 4, 7, 10, and 14 times as long as one workstation to complete the scenario for 4, 9, 14, and 19 background workstations, respectively. While none of the benchmarking programs exactly duplicated the actual performance, some prformed quite well for both individual tasks and the total scenario. The task of loading WordPerfect seemed to be the most difficult task for the background programs to represent, while they appeared to more closely simulate the file loading and saving activities.

Based on the results, the benchmark programs presented by Press generally did a good job of simulating background activity. For the configuration and the scenario used, Press's intermittent activity program best simulated actual background activity. While the LAN configuration used in the test is minimal, subsequent analyses have been conducted on different LANs utilizing performance enhanced server and client configurations. With expected performance increase, the background programs continue to simulate network activity adequately, especially for smaller configurations. (Tables of benchmark performance are available upon request.)

Timothy Paul …