System Performance: UL and BAPCo Benchmarks

Our 2022 Q4 update to the test suite for Windows 11-based systems carries over some of the standard benchmarks we have been using over the last several years, including UL's PCMark. New additions include BAPCo's CrossMark multi-platform benchmarking tool, as well as UL's Procyon benchmark suite. BPACo recently updated their SYSmark benchmark suite - while operational at a basic level, it is missing key features such as energy consumption measurement. We will start including SYSmark 30 once the open issues are resolved.

UL PCMark 10

UL's PCMark 10 evaluates computing systems for various usage scenarios (generic / essential tasks such as web browsing and starting up applications, productivity tasks such as editing spreadsheets and documents, gaming, and digital content creation). We benchmarked select PCs with the PCMark 10 Extended profile and recorded the scores for various scenarios. These scores are heavily influenced by the CPU and GPU in the system, though the RAM and storage device also play a part. The power plan was set to Balanced for all the PCs while processing the PCMark 10 benchmark. The scores for each contributing component / use-case environment are also graphed below.

UL PCMark 10 - Performance Scores

The Productivity workload benefits from the 8 high-performance Zen 3 cores in the 4X4 BOX-5800U, but the Intel systems wrest the lead in other components. Overall, the RPL-P system at default comes out on top. However, with in-band ECC enabled, the gaming workload suffers greatly. This sees the overall score for the ECC-enabled configuration to be in the middle of the pack.

UL Procyon v2.1.544

PCMark 10 utilizes open-source software such as Libre Office and GIMP to evaluate system performance. However, many of their professional benchmark customers have been requesting evaluation with commonly-used commercial software such as Microsoft Office and Adobe applications. In order to serve their needs, UL introduced the Procyon benchmark in late 2020. There are five benchmark categories currently - Office Productivity, AI Inference, Battery Life, Photo Editing, and Video Editing. AI Inference benchmarks are available only for Android devices, while the battery life benchmark is applicable to Windows devices such as notebooks and tablets. We presents results from our processing of the other three benchmarks.

UL Procyon - Office Productivity Scores

Enabling in-band ECC results in some penalty, but the scores across all MS Office workloads for both NUCS BOX-1360P/D4 configurations handily surpass the other systems.

The NUC 12 Pro Wall Street Canyon and the 4x4 BOX-5800U come out as the best bet for energy efficiency with respect to the MS Office workloads. The NUCS BOX-1360P/D4 gets a much higher score, but is let down on the energy consumption side. While not an entirely scientific metric, the Wall Street Canyon configuration delivers 733 pts / Wh, while the NUCS BOX does only 699 pts / Wh in this workload. However, we have historically seen that ASRock Industrial's BIOS for the NUC BOX lineup is rarely optimized for power consumption.

Moving on to the evaluation of Adobe Photoshop and Adobe Lightroom, we find the same pattern. Despite the 40W PL1 of the NUC 12 Pro, the 28W PL1 Core i7-1360P in the NUCS BOX is able to match it and also consume lesser energy for the workload. Enabling ECC pushes down the system towards the middle of the pack in both metrics.

UL Procyon - Photo Editing

UL Procyon evaluates performance for video editing using Adobe Premier Pro.

UL Procyon - Photo Editing

The workload takes advantage of the iGPU, which is problematic for the ECC-enabled configuration. However, the default configuration is able to come out on top by a huge margin. Being able to complete the workload faster also keeps the energy consumption low.

BAPCo CrossMark 1.0.1.86

BAPCo's CrossMark aims to simplify benchmark processing while still delivering scores that roughly tally with SYSmark. The main advantage is the cross-platform nature of the tool - allowing it to be run on smartphones and tablets as well.

BAPCo CrossMark 1.0.1.86 - Sub-Category Scores

CrossMark shows the NUCS BOX-1360P/D4 emerging on top, with the ECC-enabled configuration performing roughly on par with the NUC BOX-1260P.

Setup Notes: The In-Band ECC Option System Performance: Miscellaneous Workloads
Comments Locked

30 Comments

View All Comments

  • AntonErtl - Saturday, February 4, 2023 - link

    Given that ECC error reports are extremely rare on our systems (with typically 128GB of RAM), I don't worry about having more than one error in 512 bits. Even when a DIMM failed, it resulted in 19 ECC errors (18 uncorrectable) in 14 hours, probably noticed by scrubbing (regular walks through memory to detect whether a bit has flipped).
  • HideOut - Sunday, January 29, 2023 - link

    You linked the wrong item for barebones. You linked the old model.
  • ganeshts - Monday, January 30, 2023 - link

    Yes, the target page will get updated with the additional new model once Newegg is ready to sell them (as per the info I have from ASRock Industrial). If the URL changes, I will update it. For now, the system has just started entering the channel and is not available for end-user purchase yet.
  • GhostOfAnand - Monday, January 30, 2023 - link

    Good work, G-man. I liked the expose on this in-band ECC business. Discussion here: https://www.realworldtech.com/forum/?threadid=2104...
  • notR1CH - Tuesday, January 31, 2023 - link

    I would love to see some video tests at non-standard resolutions like "5K" (5120x2160). I have a couple of kiosks that run weird display resolutions and would love to use a NUC to power them, but with the official specs only going up to 4K I've not wanted to risk it not working.
  • abufrejoval - Saturday, February 4, 2023 - link

    Well, it stands to reason that the in-band ECC option would be physically available on all recent SoCs, because nobody can resist maximizing cut & paste IP blocks.

    But with Intel NUCs, there is literally no chance whatsoever they'd support that on non-industrial SKUs, because it's a chance to charge double.

    So I wonder if they won't force OEMs to disable that 'rogue' feature, much like AES512 or some of the recent BLK overclocking gimmicks.

    I don't even mind paying an ECC premium on hardware I run 24x7, because in those cases compromised data would cost much more. But you can't buy "ECC-variants" of common NUCs and recently even buying ECC variant mainboards of classical desktop chipsets e.g. W680 has become next to impossible, when that was relatively easy say in Haswell times.

    Don't know if it's because it's too niche or if Intel is somehow actively discouraging that market. AMD came to rescue until it went with DDR5, where ECC variants were "unobtainium".

    All I can say is that I'd be happy to throw €100 at the ability to activate in-band ECC for any SoC that phyiscally supports it, even in a notebook that might get recycled as a server after it's no longer fit for the roadtrips.
  • AntonErtl - Saturday, February 4, 2023 - link

    W680 boards are available in Germany and Austria, but they are expensive (>EUR 440).

    DDR5 UDIMMs with ECC are available in Germany and Austria from Kingston at ~EUR 210 for a 32GB UDIMM. Note that unlike for AM4, Asrock apparently no longer supports ECC with AM5, currently leaving ASUS as only supplyer for those of us who want ECC. At least AMD officially supports ECC in all socketed Ryzen 7000 CPUs up to now.
  • mode_13h - Tuesday, February 14, 2023 - link

    > Asrock apparently no longer supports ECC with AM5

    Even ASRock Rack? They've announced AM5 boards, but I'm not sure if they're yet shipping.
  • AntonErtl - Wednesday, March 1, 2023 - link

    There are no Asrock Rack AM5 boards listed on geizhals.at yet, so those may or may not support ECC when they become available.

    If this is an attempt by Asrock to get us to buy the more expensive Asrock Rack boards, it will fail: We will just buy ASUS.
  • mode_13h - Sunday, March 5, 2023 - link

    BTW, the only ECC DDR5 UDIMMs I'm seeing are all DDR5-4800. I think it'll be a while before we see anything much faster.

    This could be a point in favor of in-band ECC, if only support for it would be more widespread.

Log in

Don't have an account? Sign up now