Influence of PCI speed on RAID performance examined

Index

It's been about 9 years since the first introduction of the PCI, Vesa and OPTi bus. The introduction of these technologies meant a revolution in bus bandwidth, which was at the time limited to a mere 5MB/s from the ISA-bus. Even though Vesa Local Bus (VLB) got off to a good start it was PCI that emerged victorious. The consequences of this can be felt to this day since we are still using the same PCI technology, introduced 9 years ago, in most of our modern desktop computers. The clock speed of 33MHz, the bus bandwidth of 32-bits and the maximum bandwidth of 133MB/s has stayed the same all this time.

It won't come as a shock to you when I say the hunger for more bus bandwidth has grown dramatically over the last decade. In the early '90s a hard disk could pat him on the back for achieving a transfer rate of 3MB/s or more. Components like 8-channel audio cards, TV-cards, RAID-adapters and Gigabit-Ethernet where nowhere to be found or had an exorbitantly high price tag. Nowadays it is not a big problem to create a situation where the bandwidth of the PCI-bus is the limiting factor. For that reason server and workstation solutions have made the step to faster forms of PCI with a bus bandwidth of 64-bits and clock speeds of 66, 100 or 133MHz. Most of the time these systems are equipped with more than one PCI bus to avoid devices sharing one bus. The benefits are evident: higher bandwidth, lower latencies and fewer conflicts between devices.

Regretfully these faster PCI variants are found to be too expensive for desktop systems. A new version of the current PCI system is in the making; PCI Express. PCI Express is a completely new technology that is based on serial rather than parallel data transfer. The technology will first of all be used to replace the current AGP (Accelerated Graphics Port) port but will in the near future also replace the current PCI-bus.

One of the biggest consumers of bus bandwidth is RAID storage. Even a simple stripe-set of two 10.000rpm Serial-ATA disks can fully use the PCI-bus in some cases. On the forums we regularly hear people say a large RAID array is useless on a normal PCI-bus. They base their opinion on the fact that an array of two or more disks can already generate a higher sequential transfer rate than the PCI-bus can provide. In practice a sequential access pattern is only one of many access patterns. Opening a large file in Photoshop for instance will in the most ideal case indeed be a sequential read (the read-head does not have to move to a different part of the disk). When reading large amounts of data on a disk with heavy fragmentation there will be a greater need for extra head movements. When starting an application of which the files are further apart or simultaneously running multiple disk-intensive tasks the transfer rates will fall far below the theoretic maximum. Depending on the rpm of the drive and the performance of the actuators a change in head-position will take on average 5.5 to 12.0ms or more. Each head-movement will result in a dip in the overall transfer rate. The differences can be so dramatic that even the fastest 15.000rpm hard disks can get a transfer rate of only 4MB/s using a completely random access pattern whilst a sequential transfer rate of 75MB/s is possible.

So the question is: to what extent does the maximum transfer rate of the standard PCI-bus limit the real world performance of desktop- and server- environments. To check if there is a real loss in performance when using fast RAID-arrays combined with low bus-speeds we have tested a pair of SCSI RAID-adapters in different PCI-configurations. The purpose of this test is not only to see if the PCI-bus is a bottleneck for present day servers and workstations, but also to demonstrate if PCI Express is a useful upgrade for future desktops which will in a couple of years have the same storage-performance as do the SCSI RAID-configurations tested by us.

Testbed

The benchmarks where done on an MSI K8D Master dual Opteron board with two 32-bits 33MHz PCI-slots, three 100MHz PCI-X-slots (divided over two PCI-busses) and a 1.6GHz Opteron 242-processor. Testing was done using a Mylex AcceleRAID 600 and an LSI Logic MegaRAID Elite 1600. The Elite 1600 is an experienced dual channel Ultra160 SCSI-adapter that, in spite of his age of three years, still performs quite well. The adapter supports 64-bits 66MHz PCI and is based on a 100MHz Intel i960RN I/O processor. The Mylex AcceleRAID 600 is a modern dual channel Ultra320 SCSI-adapter which stands apart due to his high integration. The Xircon SCSI-controller and IBM PowerPC 405 I/O processor are all integrated into one chip, allowing for faster communication between these parts. Both cards where equipped with 128MB cache, the MegaRAID used 100MHz SDRAM and the AcceleRAID's flavor was 266MHz DDR SDRAM. The Mylex can set higher transfer rates than the MegaRAID due to its faster processor and memory, the latter is capable of a transfer rate of 136MB/s using write-back caching and adaptive read-ahead. Even though the MSI card is somewhat limited we felt it appropriate to use this card because of its good I/O performance, which is the most important.

The RAID-controllers where connected to four Maxtor Atlas 15k 18,4GB Ultra320 SCSI hard disks flying at 15.000 rotations per minute. These disks can achieve a maximum sequential transfer rate of 75MB/s.

MSI K8D Master mainboard and Mylex AcceleRAID 600 SCSI RAID adapter

To measure the influence of different PCI-bus speeds we tested the AcceleRAID 600 on a 100MHz PCI-X bus and on a PCI-X slot with a Promise FastTrak 100 connected to it, thus bringing the speed back to 66MHz.The MSI MegaRAID Elite 1600 was tested on a 64-bits 66MHz PCI-bus, a 32-bits 33MHz PCI-bus and a 32-bits 33MHz PCI-bus with some more load running in the background. The extra load was produced by an IOMeter transfer rate benchmark running on a Western Digital WD800JB hard disk which was connected to the Promise FastTrak. The FastTrak was placed next to the MegaRAID on the 32-bits 33MHz PCI-bus, thus generating a continuous bus-load of 25MB/s not counting overhead.

The performance of the above mentioned PCI and RAID configurations where measured using ATTO Disk Benchmark, Winbench '99 v2.0 and the StorageMark benchmarks developed in-house. These benchmarks where developed using Intel's IPEAK Storage Performance Toolkit and are based on access patterns resembling real world desktop and workstation applications. New in this review are the IPEAK SPT web- and database server benchmarks which will replace the IOMeter web server benchmarks. In hour opinion the access patterns created by IOMeter are tad to synthetic. This is why we created hour own benchmarks on a simulated Apache and MySQL server. The desktop benchmarks where done in RAID-0 and the server benchmarks in RAID-5, in both cases four disks where used.

Low-level performance

We start the benchmarking as usual; with sequential transfer rates and the average access times in Winbench 99. The cache overhead and the I/O processor restrictions do not allow the full combined sequential transfer rate of all disks to be used. In hour case the AcceleRAID will bring you up to 200MB/s whilst the MegaRAID will leave you biting dust at 77,900KB/s. PCI-clock speed reduction has little effect on both the AcceleRAID and the MegaRAID board. The MegaRAID however does slow down somewhat when the Western Digital is put to work in the background.

	Winbench 99 v2.0 - Sequential transfer rate - Begin (KB/s)
	Mylex AcceleRAID 600		PCI-X 100				197000
	Mylex AcceleRAID 600		PCI64/66				195000
	LSI MegaRAID Elite 1600		PCI64/66				77800
	LSI MegaRAID Elite 1600		PCI32/33				72800
	LSI MegaRAID Elite 1600		PCI32/33		Load		54300

	Winbench 99 - Sequential transfer rate - End (KB/s)
	Mylex AcceleRAID 600		PCI-X 100				169000
	Mylex AcceleRAID 600		PCI64/66				167000
	LSI MegaRAID Elite 1600		PCI64/66				77800
	LSI MegaRAID Elite 1600		PCI32/33				72800
	LSI MegaRAID Elite 1600		PCI32/33		Load		54500

Lowering the bus speed from 100 to 66MHz doesn't have any effect on average access times. The 0.01ms margin between the 100 and the 66MHz bus is within the error-margin. The MegaRAID shows a slight increase in access times when put on a 32-bits 33MHz PCI bus.

	Winbench 99 v2.0 - Average access time (ms)
	Mylex AcceleRAID 600		PCI-X 100				5,57
	Mylex AcceleRAID 600		PCI64/66				5,58
	LSI MegaRAID Elite 1600		PCI64/66				5,87
	LSI MegaRAID Elite 1600		PCI32/33		Load		5,98
	LSI MegaRAID Elite 1600		PCI32/33				6,03

ATTO is not usable when measuring the sequential transfer rates on RAID-controllers using cache memory. The ATTO benchmark has a maximum reach of 32MB which means the data can easily be cached by the RAM on the controllers. Which also means ATTO is very handy when testing the cache transfer rates. The graphs below will show that PCI speed has a large effect on cache transfer rates. On the MSI K8D Master motherboard with PCI-X 100MHz the AcceleRAID achieved a cache transfer rate of about 575MB/s. In the 133MHz PCI-X slot in the Iwill DP533 owned by GoT moderator BalusC it even got as high as 691,071KB/s! The Intel i960RN I/O processor used by the MegaRAID is much slower but even here lower bus speeds have a negative influence.

	ATTO Disk Benchmark - Peak read cache transfer rate (KB/s)
	Mylex AcceleRAID 600		PCI-X 133				691071
	Mylex AcceleRAID 600		PCI-X 100				578813
	Mylex AcceleRAID 600		PCI64/66				366351
	LSI MegaRAID Elite 1600		PCI64/66				136465
	LSI MegaRAID Elite 1600		PCI32/33				78951
	LSI MegaRAID Elite 1600		PCI32/33		Load		57710

LSI MegaRAID Elite 1600 - 64-bit 66MHz PCI


LSI MegaRAID Elite 1600 - 32-bit 33MHz PCI


LSI MegaRAID Elite 1600 - 32-bit 33MHz PCI under load


Mylex AcceleRAID 600 - 100MHz PCI-X


Mylex AcceleRAID 600 - 64-bit 66MHz PCI

Desktop performance

For this part of the article we used a subset of the desktop benchmarks as described in the StorageMark 2003 testing methodology (dutch). Also we used a new DVD Copy trace which was recorded while building a DVD in IfoEdit. The tests are based on access patterns of real applications and as such do a better job at representing the performance of RAID than the one-dimensional sequential transfer rate tests.

Office Light and Office Heavy are based on a trace of Business Winstone 2002. This application benchmark simulates a user performing actions in Access, Excel, FrontPage, PowerPoint, Word Microsoft Project '98, Lotus Notes, WinZip, Norton Anti-virus and Netscape Communicator. Some applications where run simultaneously in the mean while switching between programs, as a user would normally do. The Light test was recorded at minimal fragmentation and the Heavy test was recorded with heavy fragmentation and background operations. In both cases there it comes out there are big performance differences when using 64-bits 66MHz PCI and 32-bits 33MHz PCI. The regular bus is somewhere between 24 and 44 percent slower than his faster 64-bits 66MHz brother on which the MegaRAID can use all its resources.

	Tweakers.net StorageMark 2003 - Office Light (IOps)
	Mylex AcceleRAID 600		PCI-X 100				1299
	Mylex AcceleRAID 600		PCI64/66				1282
	LSI MegaRAID Elite 1600		PCI64/66				1190
	LSI MegaRAID Elite 1600		PCI32/33				935
	LSI MegaRAID Elite 1600		PCI32/33		Load		826

	Tweakers.net StorageMark 2003 - Office Heavy (IOps)
	LSI MegaRAID Elite 1600		PCI64/66				962
	Mylex AcceleRAID 600		PCI64/66				885
	Mylex AcceleRAID 600		PCI-X 100				877
	LSI MegaRAID Elite 1600		PCI32/33				813
	LSI MegaRAID Elite 1600		PCI32/33		Load		746

The workstation test is based on VeriTest Content Creation Winstone 2002. This benchmark consists of Adobe Photoshop 6.0.1, Adobe Premiere 6.0, Macromedia Director 8.5, Macromedia Dream weaver UltraDev 4, Windows Media Encoder 7, Netscape Navigator 6 and Sonic Foundry Sound Forge 5.0. Like the office benchmark there where 2 traces; a light version on an un-fragmented disk and a heavy version on a fragmented disk. The Office Heavy test uses a bigger disk area and has a WinAMP play list and a Mozilla download running in the background. During part of this test Nero is running, simulating the burning of a cd from the hard disk at four speed.

The difference in performance between 64-bits 66MHz and 32-bits 33MHz PCI are even bigger in this test compared to the Office-benchmarks. On heavily loaded PCI 32-bits 33MHz PCI-bus the MegaRAID performs a stunning 41 to 71 percent slower. The AcceleRAID 600 does not take a performance hit when using a 66MHz PCI-X bus. Remarkably the AcceleRAID performs better than the MegaRAID in Office- and Workstation light tests, but the MegaRAID outmaneuvers the AcceleRAID in the Office- and Workstation heavy tests. When using 32-bits 33MHz PCI-bus the MegaRAID is easily outperformed by the AcceleRAID on all fronts.

	Tweakers.net StorageMark 2003 - Workstation Light (IOps)
	Mylex AcceleRAID 600		PCI-X 100				1099
	Mylex AcceleRAID 600		PCI64/66				1087
	LSI MegaRAID Elite 1600		PCI64/66				943
	LSI MegaRAID Elite 1600		PCI32/33				641
	LSI MegaRAID Elite 1600		PCI32/33		Load		552

	Tweakers.net StorageMark 2003 - Workstation Heavy (IOps)
	LSI MegaRAID Elite 1600		PCI64/66				893
	Mylex AcceleRAID 600		PCI64/66				870
	Mylex AcceleRAID 600		PCI-X 100				870
	LSI MegaRAID Elite 1600		PCI32/33				704
	LSI MegaRAID Elite 1600		PCI32/33		Load		633

Software installation is a heavy test recorded while installing Adobe Photoshop 7, Microsoft Office XP and Corel WordPerfect 2000. To simulate the actions of an impatient user some smaller programs, like Nero, WinZip, Acrobat Reader, Mozilla, Home site and ACDSee where installed. The performance strongly depends on caching performance of both RAID-controller and hard disk. The MegaRAID performs admirably outperforming the AcceleRAID even on heavily loaded 32-bits 33MHz PCI-bus. The differences between a 32-bits 33MHz and a 64-bits 66MHz PCI-bus are, as before, very large. The results of the AcceleRAID prove that there is nothing wrong with the error margin of this benchmark tool, it's negligible.

	Tweakers.net StorageMark 2003 - Software Installation (IOps)
	LSI MegaRAID Elite 1600		PCI64/66				1282
	LSI MegaRAID Elite 1600		PCI32/33				962
	LSI MegaRAID Elite 1600		PCI32/33		Load		847
	Mylex AcceleRAID 600		PCI-X 100				794
	Mylex AcceleRAID 600		PCI64/66				794

Especially for RAID-benchmarks we developed a new DVD-Copy test which was recorded while creating a DVD in IfoEdit. During the test the original DVD-data was read from disk and, after editing, saved by IfoEdit. The access pattern is mostly sequential. Not surprisingly the Mylex AcceleRAID 600 is way ahead of the MegaRAID in this test. The high transfer rates make the differences between free 64-bits 66MHz and a strained legacy PCI-bus very noticeable. In this test we can also see some difference between a 66MHz and a 100MHz PCI-X bus using the AcceleRAID.

	Tweakers.net StorageMark 2003 - DVD Copy (IOps)
	Mylex AcceleRAID 600		PCI-X 100				1563
	Mylex AcceleRAID 600		PCI64/66				1493
	LSI MegaRAID Elite 1600		PCI64/66				613
	LSI MegaRAID Elite 1600		PCI32/33				452
	LSI MegaRAID Elite 1600		PCI32/33		Load		389

The weighted average of the desktop benchmarks proves that the MSI MegaRAID Elite 1600 performs on average 30.4 percent better using a 64-bits 66MHz PCI-bus instead of it's 32-bits 33MHz counterpart. Stealing 25MB/s of bus-bandwidth by the Western Digital hard disk made the difference rise to 50 percent. There is no noticeable difference in performance between 66MHz and 100MHz when the RAID-controller is using the PCI-X bus.

	Tweakers.net StorageMark 2003 - Desktop - Weighted average (IOps)
	Mylex AcceleRAID 600		PCI-X 100				1076
	Mylex AcceleRAID 600		PCI64/66				1062
	LSI MegaRAID Elite 1600		PCI64/66				964
	LSI MegaRAID Elite 1600		PCI32/33				739
	LSI MegaRAID Elite 1600		PCI32/33		Load		656

Server performance

To accurately show the performance differences we did not use Intel's IOMeter, instead we created some new server-benchmarks using the trace/playback method of Intel's IPEAK Storage Performance Toolkit. The access patterns created by IOMeter miss a sense of locality. Whilst in real life related data is often localized close to each other IOMeter creates access patterns that are always completely random. An example of locality to a high degree can be found in the form of a log file. The end of the log file will be on a fixed position on the disk which will move somewhat, but will not be placed randomly on the disk. Disk activity created by writing data to one or more log files can efficiently be optimized by a RAID-controller. This is not the case with the random patterns created by IOMeter.

The server benchmarks are based on a trace of disk-activity created by Apache2-webserver and a MySQL 4.0 database using Windows XP. The reason for using Windows is simple: Intel's IPEAK SPT does not work under non-Windows OS. The used machine was equipped with 2 AMD Opteron 242-processors and 2GB of RAM to ensure that optimizations by the OS's disk-cache had a realistic influence on disk activity. Using Apache bench Apache was ordered to serve pictures from the pics-archive of Tweakers.net which where stored five-fold to get a data-range of 2.5GB. Aside to serving pictures Apache was sent out to write some access, error and user agent logs also. The OS was given some time to fill its disk-cache to about 1.6GB before testing was started. Two types of the same test where made; a light flavor with a concurrency of 2 and a heavy flavor with a concurrency of 20 simultaneous requests. The differences in the trace can be seen; an average queue-length of 1.86 I/O's in the light test and 3.50 in the heavy test. The relation between the written and read amount of data is respectively 75/25 and 86/14. The transfer rate during the trace was 3.3MB/s on average during the light test and 35.1MB/s on average during the heavy test. Processor-load was under 100 percent in both cases which means the bottleneck here is the Mylex AcceleRAID 600 using four Atlas 15K in RAID 10.

The benchmark has a remarkable outcome; the differences between PCI-speed are very small. Even 32-bits 33MHz PCI performs almost as well as 64-bits 66MHz does under load on the MSI MegaRAID Elite 1600. The Mylex AcceleRAID 600 performs the same on both 66MHz and 100MHz PCI-X.

	Tweakers.net StorageMark 2004 - Webserver Light (IOps)
	Mylex AcceleRAID 600		PCI-X 100				435
	Mylex AcceleRAID 600		PCI64/66				435
	LSI MegaRAID Elite 1600		PCI64/66				418
	LSI MegaRAID Elite 1600		PCI32/33				413
	LSI MegaRAID Elite 1600		PCI32/33		Load		407

	Tweakers.net StorageMark 2004 - Webserver Heavy (IOps)
	LSI MegaRAID Elite 1600		PCI64/66				568
	LSI MegaRAID Elite 1600		PCI32/33				562
	LSI MegaRAID Elite 1600		PCI32/33		Load		556
	Mylex AcceleRAID 600		PCI-X 100				510
	Mylex AcceleRAID 600		PCI64/66				510

The database benchmark was made using a local copy of the Tweakers.net database. Via Apache bench a PHP-script was run, in it's turn opening random front-page, news, pricewatch and assorted using fopen(). The resulting SQL-queries made for a realistic load of the database server. During testing all logging by Apache was stopped to ensure only MySQL and Windows where accessing the hard disk. Like before we made a light and a heavy benchmark. In the light test the concurrency level in Apache bench was 10 and in the heavy test made up for 50 simultaneous requests. During this the number of queries on the database server rose to about 500 p/s which can be compared to Tweakers.net during rush-hour.

The Mylex AcceleRAID 600 and the MSI MegaRAID Elite 1600 performed completely opposite in this test. The MegaRAID is faster under a light load while the AcceleRAID takes the lead when there is a heavy load. Differences between 64-bits 66MHz and 32-bits 33MHz PCI are much bigger than in the webserver benchmarks. The MegaRAID performs 12 to 16 percent better when placed on the faster bus. When the legacy PCI-bus receives additional load the difference rises to 22 to 30 percent. In the light db-server benchmark the AcceleRAID performs the same on both the 66MHz and the 100MHz PCI-X bus, the heavy benchmarks shows an increase in speed of about 2 percent when using the 100MHz bus.

	Tweakers.net StorageMark 2004 - Database server Light (IOps)
	LSI MegaRAID Elite 1600		PCI64/66				1316
	LSI MegaRAID Elite 1600		PCI32/33				1176
	LSI MegaRAID Elite 1600		PCI32/33		Load		1075
	Mylex AcceleRAID 600		PCI-X 100				990
	Mylex AcceleRAID 600		PCI64/66				990

	Tweakers.net StorageMark 2004 - Database server Heavy (IOps)
	Mylex AcceleRAID 600		PCI-X 100				1724
	Mylex AcceleRAID 600		PCI64/66				1695
	LSI MegaRAID Elite 1600		PCI64/66				1639
	LSI MegaRAID Elite 1600		PCI32/33				1408
	LSI MegaRAID Elite 1600		PCI32/33		Load		1250

It's fair to assume that differences between legacy PCI and 64-bits 66MHz PCI-X would be even greater when the benches where unleashed on a RAID-0 array using four Atlas 15K's. Since no sober systems-engineer would trust his sensitive data to a RAID-0 array we performed the benchmarks on a RAID-5 array with four disks.

Conclusion

The benchmarks shown in previous pages clearly show that the limited bandwidth of the legacy PCI bus forms a real bottleneck for heavy RAID systems. A relatively old Ultra160 SCSI RAID-adapter performed on average 32 percent less when measured in I/O performance under desktop workloads.

Who wants to put more bandwidth-consuming devices on his legacy PCI-bus, like graphics cards, video-editing cards or multi-channel audio cards will lose something in the vicinity of 45 percent of performance. Differences are smaller in the server benchmarks, but this is partly due to the usage of RAID-5 instead of RAID-0.

Tweakers who want a big RAID-array are wise to buy a motherboard that is equipped with a 64-bits PCI-bus. Besides the advantage of more bandwidth the PCI-X mainboards also enable you to use separate PCI-busses thus eliminating conflicts between for instance a sound card and a RAID-adapter. Unfortunately 64-bits PCI-slots are mostly found on expensive workstation mainboards with come with a price tag of about 450 dollars. The cheapest solution for a high-bandwidth system is buying a dual Athlon motherboard based on the AMD 760MPX chipset. These boards can be obtained by forking over about 250 dollars, unless you can get one secondhand. Note that this boards have to do without a lot of features present on modern mainboards, like USB2.0, FireWire and on-board Gigabit Ethernet.

The majority of desktop users will find the prices of workstation class mainboards to be too steep. They will just have to wait for PXI-Express to arrive. This technology promises to be a lot more scalable than the current PCI standard which is still stuck on a maximum bandwidth of 133MB/s. Hopefully the availability of bandwidth will be higher than the demand for it, making bandwidth limitations a thing of the past. Until then manufacturers are trying to bring some relief by connecting (Serial) ATA-RAID and Gigabit Ethernet controllers directly with the Southbridge-interconnect taking away the need to use valuable PCI-bandwidth.

We thank both AMD and MSI for making this review possible by providing us with respectively two Opteron processors and the K8D Master dual Opteron mainboard. Finally we would like to thank original article from dutch to english.