Storage Review has acquired a good reputation with its reviews and tests of hard drives. Actually, SR is the only website performing serious investigations considering real world performance of hard drives. The site did some pioneering work on developing methods for testing hard drives and was the first to benchmark hard drives using the IPEAK Storage Performance Toolkit. Other sites like Tweakers.net and AnandTech followed SR's lead. However, one area of interest that Storage Review missed was RAID, adapters and configurations. The article about TCQ and RAID that was published last month does not seem to change that trend, despite its disputed conclusions.
Storage Review's current test suite has reached the age of two since it was introduced in 2002, built on Office DriveMark 2002, High-End DriveMark 2002, Bootup DriveMark 2002, Gaming DriveMark 2002 and Server DriveMark 2002. Server tests are based on some synthetically created access patterns, created by IOMeter (which relevance we already doubted here, links to a Dutch page) and desktop testing is performed by IPEAK SPT. Office DriveMark 2002 is based on a trace during half an hour of office usage by Storage Review's webmaster. High-End DriveMark 2002 is a trace of disk usage during Content Creation Winstone 2001, Bootup DriveMark 2002 speaks for itself and Gaming DriveMark 2002 is a trace of activities performed by games like Black & White, Counterstrike, Diablo 2 and The Sims.
Storage Review, unlike AnandTech, does give some information about its traces in the description of their testing methods. It seems that the Office DriveMark 2002 trace had an average queue depth of 1,34 I/O's and an average of 23KB transfer size. The Content Creation Winstone 2001 trace on average contains 1,4 I/O's with a transfer size of 69,5KB. Compared to statistics of our more modern Winstone 2004 traces, both Business Winstone 2004 and Multimedia Content Creation 2004 seem to have higher loads than Storage Review's corresponding tests. BSW2004, with some extra Winamp and eMule activity, gives us a queue with an average of 3,22 I/O's and MCCW2004 even has 8,82 I/O's in its queue. The transfer sizes seem to be in line with those of Storage Review: 25,7KB in the BSW2004 trace and 62,1KB in MCCW2004's trace.
This higher queue-load in Tweakers.net's traces is positive for RAID 0 configurations, which can easily distribute commands to different disks when facing a large queue. Storage Review's tests on the contrary offer almost no possibility for simultaneous transactions, which puts a limit on performance gains with a RAID 0 array, since all improvements should come from higher transfer rates with sequential access. The fact that traces made with Business Winstone 2004 and Multimedia Content Creation 2004 provide quite different results with Storage Review's older traces, indicate clearly that our colleagues should actualize their benchmarks. BSW2004 and MCCW2004 are both realistic simulations of multi-tasking users in some current software-editions. Disk activities generated by the Winstone 2004 suite are certainly no exception for target group.
The hypothesis that RAID 0 will perform poorly in desktop benchmarks by Storage Review is confirmed in a recent article by Storage Review, on the topic of TCQ and RAID performance of the Raptor 740GD. Using a Promise FastTrak S150 TX4 controller, two Raptors show a pitiful improvement of 10,3 percent in Office Drivemark 2002. The performance gain in High-End DriveMark 2002 reaches a sad 12,9 percent. Only the Windows XP boot benchmarks show a decent improvement of 46,2 percent.
 | ||||||
 | | |||||
| ||||||
| Tweakers.net | | Promise S150 TX2plus | |    36,1 | |
| ||||||
| Storage Review | | Promise S150 TX4 | |    10,3 | |
| ||||||
| ||||||
| | |||||
| ||||||
| Tweakers.net | | Promise S150 TX2plus | | 50,0 | |
| ||||||
| Storage Review | | Promise S150 TX4 | | 12,9 | |
| ||||||
| ||||||
| | |||||
| ||||||
| Tweakers.net | | Promise S150 TX2plus | | 45,6 | |
| ||||||
| Storage Review | | Promise S150 TX4 | | 46,2 | |
| ||||||
Performance scaling in the office and content creation benchmarks performed by Storage Review contrast severely with Tweakers.net's tests, where the Raptor array, as mentioned earlier, showed a 36,1 percent improvement in Business Winstone 2004 and a nice gain of 50 percent in the Multimedia Content Creation Winstone 2004. The fact that Storage Review had the FastTrak S150 TX4 crawling on a 32-bits 33MHz PCI-bus can't be a fair explanation for the disappointing results, since our FastTrak S150 TX2plus was also installed on a legacy PCI-slot. Bandwidth is of greater importance for RAID 0 performance, since the improvements in Storage Review's tests had to be obtained with faster sequential transfer rates. Differences between the Promise FastTrak S150 TX4 and the TX2plus can in no possible way cause such a big difference in results, since both adapters are very similar to each other.
The fact that there is a significant improvement in the Windows XP boot benchmark clearly shows that Storage Review's system could have some obvious advantages with RAID 0 when using correct workloads. Therefore, we suggest that Storage Review's benchmarks are not typical for the demanding use patterns in current office and content creation applications.