Searching for the Memory Holy Grail - Part 2

Test Design

One of the difficulties in testing memory is that most of the memory benchmarks available are synthetic. While synthetic benchmarks can be useful in comparing performance, they can also paint a distorted picture of real-world performance. This is the reason why AnandTech has always preferred benchmarking with real applications. Benchmarks using games are dependent on many system components for their results, including the CPU speed and Video Card, which have a major impact on the final scores. While memory does impact the game benchmark score, it is only one small part of the total score. Finding a game benchmark that is sensitive to memory is not always easy. We discovered Gun Metal 2, for instance, tends to be video-card bound, making it very useful for testing video cards, but not so useful for measuring subtle differences in system performance. After looking at available game benchmarks, we found Quake3 and Unreal Tournament 2003 to be the most useful for our memory testing.

The following Benchmarks were used in our Memory Testing:

1) SiSoft Sandra Max3 UNBuffered Memory Test

Part 1 of “Searching for the Memory Holy Grail” demonstrated the usefulness of the SiSoft Sandra UNBuffered Memory Test as a sensitive benchmarking tool for memory bandwidth. The Sandra UNBuffered Memory Test turns off Memory Buffering schemes in an attempt to improve the measure of raw memory bandwidth. As a result, it also correlates well with bandwidths reported with Memtest86, an industry-standard memory testing tool.

The idea of the UNBuffered Memory Benchmark is very simple — you merely turn-off all memory buffering techniques. Sandra makes this very easy to do. Select “Memory Benchmark”, right-click “Module Options”, and uncheck the nine boxes that are related to buffering.

2) SiSoft Sandra Max3 Standard Memory Test

The UNBuffered Memory Benchmarks are quite different from what you may be accustomed to seeing in memory testing with SiSoft Sandra. For reference, we are again including the Sandra Max3 standard Memory Test, sometimes called the Buffered Memory Test.

3) Super PI

Pure number-crunching benchmarks are very useful for measuring system bandwidth. Some of the more popular number-crunchers are the MPEG/DIVX encoding tests, such as the ones that we used in our standard motherboard testing, and Super PI. MPEG/DIVX tests are valuable for a single motherboard benchmark and in cross-platform testing — Athlon vs. Pentium4, for example. However, they are often very sensitive to the test environment or system configuration, and can be difficult to use reliably in an environment that tests a large number of conditions with the same test, such as we will be doing here in our memory testing. Super PI, on the other hand, is very simple to use and has been shown to be less sensitive to the operating system environment. In other words, we don’t have to reinstall the operating system on a clean hard drive each time we run a benchmark just to get reliable numbers.

Super PI for Windows 1.1 is a freeware program developed by the Super Computer Consortium at the University of Tokyo. The concept of Super PI is very simple — it calculates the value of pi to “x” number of places, and reports the time this calculation requires. We chose to use 2 million places in our tests. Super PI measures total system bandwidth, and memory is only part of that bandwidth, since the CPU has a significant impact on results. We therefore would expect to see smaller changes in Super PI relative to larger changes in memory-only benchmark tests like Sandra.

4) Quake3 Demo FOUR.dm_66

Quake 3 Demo FOUR is one of our standard game benchmarks. As Evan Lieb showed in his PC3200 memory tests, Quake3 can also reveal variations in memory performance. You will likely be surprised how sensitive Quake3 can actually be in testing wide variations in Memory Speed. We run the benchmark three times, check for score consistency, repeat if we see any wide variation in individual scores, and then average the three scores for the reported Frames per Second (FPS) value.

5) Unreal Tournament 2003 Demo

The Benchmark program built into the UT2003 demo is a contemporary game test that does respond to variations in memory bandwidth. We used it mainly to show the impact of memory speed on UT2003 scores, and to confirm the validity of Quake3 as a real-world test of memory performance. With our new standard ATI 9800 PRO video card, UT2003 shows variation in both Flyby and Botmatch in tests with memory of different speeds. All benchmarks are run at our standard 1024x768 resolution.

Motherboard, CPU, and Peripherals

In Part 1 of “Searching for the Memory Holy Grail”, we used the Abit IC7 for our 875 tests and the Asus P4P800 Deluxe for 865 tests. Because of the impact of PAT on/off at different FSB, we decided to use only the Intel 875 for testing in Part 2. This presented our first problem, since the Abit IC7 has a strange quirk in the way it handles 1:1 memory. No matter what we did, we could not operate the Abit IC7 at greater than a 255 setting (1020FSB). We talked with Abit about this issue, and they are hard at work on a BIOS revision to correct this problem. To be fair, most users will not need to run memory at faster than a setting of 255, and 5:4 and 3:2 memory ratios do not have this issue. However, for our tests here, we expected that we might reach a speed of 275 (1100FSB), and the 255 limitation was not acceptable.

Next, we considered the DFI 875PRO LanParty as the motherboard for our testbed. The DFI had no problem handling settings above 255, which corrected that problem. However, as Evan Lieb pointed out in his review of the 875PRO, the vDIMM range to only 2.7V was too limiting for our high-speed memory tests. DFI has told us that they are releasing an updated version of the 875PRO in the near future with expanded vDIMM options. If that were available today, the DFI would have worked well for our testbed.

The latest revision of the ASUS flagship 875 motherboard is the ASUS P4C800-E. This board adds Intel GigaLAN (using the dedicated Intel CSA bus), and incorporates the ICH5R Southbridge with Intel SATA RAID. We will be doing a review update on this new revision of the ASUS flagship Canterwood shortly. The P4C800-E met our requirements of high speed 1:1 operation and a vDIMM adjustment range that was useful. vDIMM is available to 2.85V on the P4C800-E. As a bonus, we were able to use Intel SATA RAID with SATA drives for all testing.

Our 3.0C Pentium 4 800FSB chip was not very useful for testing DDR500 memory. With a maximum overclock of around 245 (980FSB), we could not even reach the rated speed of the memory. We settled on a 2.4C 800FSB Pentium 4 that has been proven to perform very well at high speeds. On the ASUS P4C800-E, this 2.4C was able to reach a stable 288 setting (1152FSB) at default 1.525V, and a setting of 298 (1192FSB) with a modest vCore setting of 1.6V. We were confident that this test setup would allow us to reach the maximum speeds possible with memory rated at a high as DDR500, since we did not anticipate that synchronous operation would exceed DDR596 in our testing.

Since all testing would be done on a single testbed configuration and only compared to test results on that testbed, we chose the best-performing components that we had available. For CPU cooling, we used the Thermalright SLK-900U heatsink with a 120mm adjustable-speed Vantec Tornado cooling fan. The idea here was to remove any concerns about CPU cooling or overclocking ability from the memory test as much as possible. For the video card, we used our new standard ATI Radeon 9800 PRO with 128 MB memory. Hard drives were a pair of Western Digital Raptor 10,000RPM Serial ATA drives running in a SATA RAID 0 (Striping) configuration on the stock Intel ICH5R.