Comments on the latest phoronix benchmark

[Yonas Yanfa]

Hi,

Link: https://www.phoronix.com/review/bsd-linux-threadripper-7980x/3

The following six benchmarks show FreeBSD 14.1 performed poorly compared
to other OSes:

[ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs
15,267 (CentOS Stream 9)
[ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779
(FreeBSD) vs 2,224 (Ubuntu 24.04 LTS)
[ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD)
vs 8,253,203 (Ubuntu 24.04 LTS)
[ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs
3,557,329 (NetBSD)
[ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465
(FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS)
[ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD)
vs 313,648,228 (DragonFlyBSD)

Does anyone know why, and how we can improve the numbers?

Cheers,
Yonas



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Warner Losh]

Post by Yonas Yanfa
Hi,
Link: https://www.phoronix.com/review/bsd-linux-threadripper-7980x/3
The following six benchmarks show FreeBSD 14.1 performed poorly compared
[ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs
15,267 (CentOS Stream 9)
[ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779
(FreeBSD) vs 2,224 (Ubuntu 24.04 LTS)
[ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD)
vs 8,253,203 (Ubuntu 24.04 LTS)
[ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs
3,557,329 (NetBSD)
[ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465
(FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS)
[ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD)
vs 313,648,228 (DragonFlyBSD)
Does anyone know why, and how we can improve the numbers?

Stress-ng is not intended to be a benchmark (and says so in its docs) and
does all kinds of special things on Linux only. It has a bunch of stubs on
systems that didn’t implement something. It's a deeply flawed. I believe
this information is in the comments to the article.

That said, there are speed improvements we can make to things, like our VM
that other benchmarks do show issues with... but first the benchmarks need
to actually be apples to apples comparisons.

Warner

Cheers,

Post by Yonas Yanfa
Yonas

[Kyle Taylor]

I've been working on a neutral numerical benchmarking interface that uses
Python/R. I haven't thought of using it to compare different versions of
FreeBSD. I wrote it to compare FreeBSD/Linux for compute intensive
workflows. But it would be interesting to compare different versions. I
rarely expect the differences between OS versions to be that different, but
perhaps that's a mistake.

Folks at Klara (that do a lot of impressive benchmarking with FreeBSD)
always recommend writing your own tests to capture performance data. Your
workloads could be web traffic, databases, compute, etc... but it very
likely will not look much like the pre-canned tests that Phoronix puts
together. I do look at Larabel's test results on BSD. I follow it with
interest, even. But I tend to treat it as early warning testing. I'm
usually left trying to figure out a poor result for FreeBSD on my own. In
my own CPU/RAM/Filesystem benchmarking workflows on standard installations,
FreeBSD and Linux are often very close.

Best - Kyle


[1]
https://klarasystems.com/articles/evaluating-freebsd-current-for-production-use/
[2] https://the-integral.dev/post/freebsd-for-data-science/

Post by Warner Losh

Post by Yonas Yanfa
Hi,
Link: https://www.phoronix.com/review/bsd-linux-threadripper-7980x/3
The following six benchmarks show FreeBSD 14.1 performed poorly compared
[ 4.3x slower ] Stress-NG 0.17.08: Socket Activity -- 3,569 (FreeBSD) vs
15,267 (CentOS Stream 9)
[ 2.9x slower ] Stress-NG 0.17.08: Glibc Qsort Data Sorting -- 779
(FreeBSD) vs 2,224 (Ubuntu 24.04 LTS)
[ 2.2x slower ] Stress-NG 0.17.08: AVX-512 VNNI -- 3,626,943 (FreeBSD)
vs 8,253,203 (Ubuntu 24.04 LTS)
[ 1.5x slower ] Stress-NG 0.17.08: CPU Cache -- 2,322,478 (FreeBSD) vs
3,557,329 (NetBSD)
[ 1.5x slower ] Stress-NG 0.17.08: Fused Multiply-Add -- 63,639,465
(FreeBSD) vs 96,258,730 (Ubuntu 24.04 LTS)
[ 1.3x slower ] Stress-NG 0.17.08: Semaphores -- 230,741,240 (FreeBSD)
vs 313,648,228 (DragonFlyBSD)
Does anyone know why, and how we can improve the numbers?

Stress-ng is not intended to be a benchmark (and says so in its docs) and
does all kinds of special things on Linux only. It has a bunch of stubs on
systems that didn’t implement something. It's a deeply flawed. I believe
this information is in the comments to the article.
That said, there are speed improvements we can make to things, like our VM
that other benchmarks do show issues with... but first the benchmarks need
to actually be apples to apples comparisons.
Warner
Cheers,

Post by Yonas Yanfa
Yonas