************************************************************************ What's New in QBench 1.3? -------------------------- Before describing what is different in version 1.3 of QBench, first note that the basic structure of QBench measurements and calculations has not changed. All results obtained using version 1.3 are directly comparable with results obtained using previous versions. Two new command line options have been added, and the drive's data transfer rate is calculated and presented in more detail than in previous versions. The test methods, calculations, and even the help files and documentation have not changed from the originally released version 1.21. Only this file, QB130NEW.TXT, documents the changes in version 1.3. This file only includes a description of the enhancements found in QBench 1.3. No attempt is made to provide an introductory explanation of the workings of QBench. Please refer to the original program help files and the QBINSTR user's manual file for more information. The enhancements for version 1.3 of QBench are: (1) Additional data transfer rate calculations have been added, reducing the reliance on data access time as the single measure of disk performance. (2) A new option which allows QBench to be run from a batch file without operator intervention. (3) An advanced option which provides more detail on the statistical calculations used to obtain the data access time and the data transfer rate. ************************************************************************ (1) The Increasing Importance of Data Transfer Rate ------------------------------------------------------- The initial release of QBench emphasized data access time as the primary metric of disk drive performance. There were two main reasons for this choice. First, data access time is the most accurate single indicator of a drive's DOS performance, since DOS transfers are typically very short and thus access time is the primary component of these transfers. Additionally, a drive's data access time is generally independent of the speed of the machine on which the test is run, and thus it is a good indicator of drive performance, even when the drive is moved from one system to another. Data transfer rate, as used here, is defined as the rate at which data is transferred, once it has been accessed. Since the relationship between the size of a data transfer (in sectors) and the command service time required to complete that transfer (in ms) is linear and may be plotted as a straight line, a drive's performance characteristics may be completely described by the slope (in ms/sector) and intercept (in ms) of that line. The data access time, as reported by QBench, is approximately equal to the intercept of this line. The data transfer rate used by QBench is the reciprocal of the slope, with the units converted from sectors/ms to the more traditional KBytes/second. Other benchmarks which report transfer rate typically measure the total time required to complete a data transfer command. This quantity includes not only transfer time, but also additional fixed overhead per command such as command overhead in the computer, drive access time, etc. Thus the transfer rate reported is highly dependent upon the length of the data transfer used in the test (how much data gets transferred for each command processed), and a different transfer rate is reported for each transfer size. QBench's data transfer rate measure has the advantage of being completely independent of the transfer size. Note that any transfer rate measure, including QBench's data transfer rate, is highly system dependent, and will vary greatly due to differences in CPU speed and I/O bus implementation. However, if different drives are measured on the same computer system, the drive- dependent aspects of data transfer rate can be properly observed. Why does the new version of QBench increase the emphasis on data transfer rate as a measure of performance in addition to data access time? There are several reasons. Although this measure is system dependent, it is an important component of the overall performance of a computer's I/O subsystem. Also, for operating systems other than DOS which transfer data in units longer than a sector or two, the data transfer rate accounts for an increasing proportion of the overall command service time. The final reason has to do with the method QBench uses to calculate data access time. Since data access time cannot be measured directly, it is calculated in QBench by subtracting the data transfer time from the command service time for a single sector transfer. For a given single sector command service time, a drive with a poor data transfer rate may appear to have a better data access time than it should. In summary, data access time and data transfer rate are intimately connected, and both measures should be reported to properly describe the performance of a hard disk. Just remember that the data transfer rate is system dependent, and can only be understood in the context of the machine on which the benchmark is run. An example of the differences in QBench output between the old and new versions is presented below. The lines below are found at the end of the "Detailed Results" section. QBench 1.21 output: RESULTS Sequential Random Weighted Read Write Read Write Average Access Time: 0.47 ms 0.32 ms 16.94 ms 22.62 ms 6.99 ms* Transfer Rate KByte/Second: 956 ______________________________________________________________________ * Data Access Time is 7.0 ms ______________________________________________________________________ QBench 1.3 output: RESULTS Sequential Random Weighted Read Write Read Write Average Access Time 0.47 ms 0.32 ms 16.94 ms 22.62 ms 6.99 ms* Transfer Rate 859 989 949 1304 956 ** (KB/Sec) ______________________________________________________________________ * Data Access Time is 7.0 ms ______________________________________________________________________ ** Data Transfer Rate is 960 KB/sec ______________________________________________________________________ The weighted average value (which is also the data transfer rate) in version 1.3 is the same as the value reported as Transfer Rate in version 1.21. In addition, version 1.3 provides detail on the contribution to the data transfer rate for each type of disk access. Finally, the data transfer rate (rounded to the nearest 10 KB/sec to better represent the accuracy of the measurement) is displayed at the bottom along with the data access time. One final note: the data access time is equal to the weighted average (using the standard arithmetic mean) of the individual access times for each type of disk access, using the workload percentages specified in the program as weights. For data transfer rate, the weighted harmonic mean of the individual transfer rates is used. ************************************************************************ (2) Running QBench From a Batch File ------------------------------------- The /O or Output option (when used in conjunction with the previously available /D option) allows QBench to be run from a batch file with the Detailed Results for the test run automatically stored in a file for later analysis and comparison. This relatively minor enhancement can be a major improvement when a number of drives are being tested or a number of runs are desired for a particular drive, since all of the testing can be done in an automated manner without operator intervention. An example batch file might include the following lines: QBENCH /D:1 /O:RUN1.DAT QBENCH /D:1 /O:RUN2.DAT QBENCH /D:1 /O:RUN3.DAT These three lines would cause QBench to be run three times for the first drive in the system, with the results of the three runs stored in the files RUN1.DAT, RUN2.DAT and RUN3.DAT. Note that the /D option must be used in conjunction with /O for batch file operation to indicate which of the system drives QBench should test. ************************************************************************ (3) Additional Detail on Statistical Calculations -------------------------------------------------- Please note before reading this section that the average user will NOT want the additional detail provided by this option. The /S option allows a user with a background in statistics to understand in detail the least squares curve fitting calculations used internally by QBench. If QBench is invoked with the /S command line option, three lines of additional output will be included at the end of the Detailed Results as shown in the example below. QBench 1.3 output (with /S option): RESULTS Sequential Random Weighted Read Write Read Write Average Access Time 0.47 ms 0.32 ms 16.94 ms 22.62 ms 6.99 ms* Transfer Rate 859 989 949 1304 956 ** (KB/Sec) ______________________________________________________________________ * Data Access Time is 7.0 ms ______________________________________________________________________ ** Data Transfer Rate is 960 KB/sec ______________________________________________________________________ Intercept 1.145 ms 0.368 ms 17.085 ms 20.871 ms 7.052 ms Slope 0.582 ms 0.506 ms 0.527 ms 0.384 ms 0.523 ms R^2 0.998 1.000 0.999 0.979 0.999 The additional three lines show the slope and intercept of the least- squares regression line, while R^2 ("R squared") is the square of the sample correlation coefficient (sometimes called the coefficient of determination) and shows the degree of linear fit of the data (1.000 indicates a perfect linear fit). Please consult a statistics textbook for further information on these quantities. ************************************************************************