Searching for the Memory Holy Grail - Part 2
by Wesley Fink on August 26, 2003 11:11 PM EST- Posted in
- Memory
Test Design
One of the difficulties in testing memory is that most of the memory benchmarks available are synthetic. While synthetic benchmarks can be useful in comparing performance, they can also paint a distorted picture of real-world performance. This is the reason why AnandTech has always preferred benchmarking with real applications. Benchmarks using games are dependent on many system components for their results, including the CPU speed and Video Card, which have a major impact on the final scores. While memory does impact the game benchmark score, it is only one small part of the total score. Finding a game benchmark that is sensitive to memory is not always easy. We discovered Gun Metal 2, for instance, tends to be video-card bound, making it very useful for testing video cards, but not so useful for measuring subtle differences in system performance. After looking at available game benchmarks, we found Quake3 and Unreal Tournament 2003 to be the most useful for our memory testing.
The following Benchmarks were used in our Memory Testing:
1) SiSoft Sandra Max3 UNBuffered Memory Test
Part 1 of “Searching for the Memory Holy Grail” demonstrated the usefulness of the SiSoft Sandra UNBuffered Memory Test as a sensitive benchmarking tool for memory bandwidth. The Sandra UNBuffered Memory Test turns off Memory Buffering schemes in an attempt to improve the measure of raw memory bandwidth. As a result, it also correlates well with bandwidths reported with Memtest86, an industry-standard memory testing tool.The idea of the UNBuffered Memory Benchmark is very simple — you merely turn-off all memory buffering techniques. Sandra makes this very easy to do. Select “Memory Benchmark”, right-click “Module Options”, and uncheck the nine boxes that are related to buffering.
2) SiSoft Sandra Max3 Standard Memory Test
The UNBuffered Memory Benchmarks are quite different from what you may be accustomed to seeing in memory testing with SiSoft Sandra. For reference, we are again including the Sandra Max3 standard Memory Test, sometimes called the Buffered Memory Test.
3) Super PI
Pure number-crunching benchmarks are very useful for measuring system bandwidth. Some of the more popular number-crunchers are the MPEG/DIVX encoding tests, such as the ones that we used in our standard motherboard testing, and Super PI. MPEG/DIVX tests are valuable for a single motherboard benchmark and in cross-platform testing — Athlon vs. Pentium4, for example. However, they are often very sensitive to the test environment or system configuration, and can be difficult to use reliably in an environment that tests a large number of conditions with the same test, such as we will be doing here in our memory testing. Super PI, on the other hand, is very simple to use and has been shown to be less sensitive to the operating system environment. In other words, we don’t have to reinstall the operating system on a clean hard drive each time we run a benchmark just to get reliable numbers.Super PI for Windows 1.1 is a freeware program developed by the Super Computer Consortium at the University of Tokyo. The concept of Super PI is very simple — it calculates the value of pi to “x” number of places, and reports the time this calculation requires. We chose to use 2 million places in our tests. Super PI measures total system bandwidth, and memory is only part of that bandwidth, since the CPU has a significant impact on results. We therefore would expect to see smaller changes in Super PI relative to larger changes in memory-only benchmark tests like Sandra.
4) Quake3 Demo FOUR.dm_66
Quake 3 Demo FOUR is one of our standard game benchmarks. As Evan Lieb showed in his PC3200 memory tests, Quake3 can also reveal variations in memory performance. You will likely be surprised how sensitive Quake3 can actually be in testing wide variations in Memory Speed. We run the benchmark three times, check for score consistency, repeat if we see any wide variation in individual scores, and then average the three scores for the reported Frames per Second (FPS) value.
5) Unreal Tournament 2003 Demo
The Benchmark program built into the UT2003 demo is a contemporary game test that does respond to variations in memory bandwidth. We used it mainly to show the impact of memory speed on UT2003 scores, and to confirm the validity of Quake3 as a real-world test of memory performance. With our new standard ATI 9800 PRO video card, UT2003 shows variation in both Flyby and Botmatch in tests with memory of different speeds. All benchmarks are run at our standard 1024x768 resolution.
Motherboard, CPU, and Peripherals
In Part 1 of “Searching for the Memory Holy Grail”, we used the Abit IC7 for our 875 tests and the Asus P4P800 Deluxe for 865 tests. Because of the impact of PAT on/off at different FSB, we decided to use only the Intel 875 for testing in Part 2. This presented our first problem, since the Abit IC7 has a strange quirk in the way it handles 1:1 memory. No matter what we did, we could not operate the Abit IC7 at greater than a 255 setting (1020FSB). We talked with Abit about this issue, and they are hard at work on a BIOS revision to correct this problem. To be fair, most users will not need to run memory at faster than a setting of 255, and 5:4 and 3:2 memory ratios do not have this issue. However, for our tests here, we expected that we might reach a speed of 275 (1100FSB), and the 255 limitation was not acceptable.Next, we considered the DFI 875PRO LanParty as the motherboard for our testbed. The DFI had no problem handling settings above 255, which corrected that problem. However, as Evan Lieb pointed out in his review of the 875PRO, the vDIMM range to only 2.7V was too limiting for our high-speed memory tests. DFI has told us that they are releasing an updated version of the 875PRO in the near future with expanded vDIMM options. If that were available today, the DFI would have worked well for our testbed.
The latest revision of the ASUS flagship 875 motherboard is the ASUS P4C800-E. This board adds Intel GigaLAN (using the dedicated Intel CSA bus), and incorporates the ICH5R Southbridge with Intel SATA RAID. We will be doing a review update on this new revision of the ASUS flagship Canterwood shortly. The P4C800-E met our requirements of high speed 1:1 operation and a vDIMM adjustment range that was useful. vDIMM is available to 2.85V on the P4C800-E. As a bonus, we were able to use Intel SATA RAID with SATA drives for all testing.
Our 3.0C Pentium 4 800FSB chip was not very useful for testing DDR500 memory. With a maximum overclock of around 245 (980FSB), we could not even reach the rated speed of the memory. We settled on a 2.4C 800FSB Pentium 4 that has been proven to perform very well at high speeds. On the ASUS P4C800-E, this 2.4C was able to reach a stable 288 setting (1152FSB) at default 1.525V, and a setting of 298 (1192FSB) with a modest vCore setting of 1.6V. We were confident that this test setup would allow us to reach the maximum speeds possible with memory rated at a high as DDR500, since we did not anticipate that synchronous operation would exceed DDR596 in our testing.
Since all testing would be done on a single testbed configuration and only compared to test results on that testbed, we chose the best-performing components that we had available. For CPU cooling, we used the Thermalright SLK-900U heatsink with a 120mm adjustable-speed Vantec Tornado cooling fan. The idea here was to remove any concerns about CPU cooling or overclocking ability from the memory test as much as possible. For the video card, we used our new standard ATI Radeon 9800 PRO with 128 MB memory. Hard drives were a pair of Western Digital Raptor 10,000RPM Serial ATA drives running in a SATA RAID 0 (Striping) configuration on the stock Intel ICH5R.
77 Comments
View All Comments
Anonymous User - Thursday, August 28, 2003 - link
I am tired of setting the memory timing and bench mark all the time. Is there a program there which can tell me what kind of results I would get? Say if I can increase my CPU by 5 MHz but have to set back my memory timing a bit, which way should I go?oldfart - Thursday, August 28, 2003 - link
Here are some reviews comparing tight timings Vs loose:http://www.hardtecs4u.com/reviews/2003/ddr400_roun... (need language translator)
http://www.octools.com/index.cgi?caller=articles/c...
http://www.3dxtreme.org/Corsair_xms3700_twinx_p1.s...
Anonymous User - Thursday, August 28, 2003 - link
I think this was an extremely helpful and thorough review. There was one comparison, though, that I would find most helpful and haven't found anywhere. I'm currently debating the importance of running synchronously, and thus found the section "Does memory speed really matter in the real world" extremely interesting. However, I would have greatly preferred one additional test -- running 1066FSB at 3:2 and 5:4 with memory with tight timings (2-2-2-5), since my real debate is whether to buy PC3200 or PC3500 with tight timings and run at 5:4 or 3:2, or PC4000 with loose timings and run at 1:1. While I expect that the synchronous memory would result in better performance, I'd really like to know how much better, since PC4000 memory is expensive!Thanks,
Steve
Dennis Travis - Thursday, August 28, 2003 - link
Great Review Wes. Keep Em coming. I am not "PAID" to say this either. I wanted to. I am getting nothing for it either. Just the satisfaction of telling Wes I loved his review.Anonymous User - Thursday, August 28, 2003 - link
Great review Wesley. Nah I'm not paid to say this, I just enjoyed the review!Anonymous User - Thursday, August 28, 2003 - link
I have tested Kingston HyperX RAM at 1:1 3-4-4-8 @ DDR500, and 5:4 2-2-2-5 @ DDR400 at the same FSBs on a P4P800, with MAM Enabled and Turbo performance mode in both cases. While the 1:1 gets about 3-5% better Sandra bandwidth scores (buffered and unbuffered), SuperPI completes about 1.5% sooner at the 5:4 settings.So real-world performance may be slightly better at 5:4, but you won't win any Sandra bragging rights with it.
--MeowChow
Anonymous User - Thursday, August 28, 2003 - link
oldfart - Wednesday, August 27, 2003 - link
Wow, looks I'm not the only guy who wants to see250 FSB
1:1 3-4-4-8
5:4 2-2-2-6
type of testing. I've seen several reviews that show the lower latency ram @ 5:4 to be faster.
Part 3??
Anonymous User - Wednesday, August 27, 2003 - link
Hey Wesley,Thanks for all the good info...
Any chance you could test Various FSB's
5:4 2-2-2-5 vs the same FSB at 1:1 2.5-4-4-7
It would be great to show the readers how the new PC4000 REALLY compares to older slower low latency RAM, Mushkin PC3500 level2 would be perfect for that.
Now that would be a seriously good Anandtech caliber review. :D
Wesley Fink - Wednesday, August 27, 2003 - link
#32 and #40 -Mushkin did not have a product in our hands when the review was done. In fact I completed a review of Mushkin PC3500 Level II just a couple of days ago, and compared it's performance to ALL the memory in this review at DDR400. I also tested Adata DDR450, which did not meet our requirement of running at DDR500, but DID perform well at DDR400.
The reviews should be up here shortly. The Mushkin did VERY well at DDR400 to DDR450. Mushkin is also about to release DDR500 - but they did not have a product ready in time for our review. We WILL be testing it as soon as it is available if time allows.