polling interval

[void]

I dimly remember a HZ setting, maybe kernel config option?
From years ago. I can't find it in recent /sys/amd64/conf/*
Does this functionality still exist or has it moved elsewhere?
Or did it ever exist?

I am seeing slow network throughput speeds in virtio-net relative to linux vms
on the same host and thought maybe it had something to do with polling too fast.

[Tomoaki AOKI]

On Mon, 14 Oct 2024 12:47:27 +0100

Post by void
I dimly remember a HZ setting, maybe kernel config option?
From years ago. I can't find it in recent /sys/amd64/conf/*
Does this functionality still exist or has it moved elsewhere?
Or did it ever exist?
I am seeing slow network throughput speeds in virtio-net relative to linux vms
on the same host and thought maybe it had something to do with polling too fast.
--

Tunable kern.hz? Or something else?

[Dag-Erling Smørgrav]

Post by void
I dimly remember a HZ setting, maybe kernel config option?
From years ago. I can't find it in recent /sys/amd64/conf/*

Look in sys/conf/NOTES.

DES

[void]

Post by Tomoaki AOKI
Tunable kern.hz? Or something else?

I think that may be it!

# sysctl kern.hz
kern.hz: 100

# sysctl kern.hz_max
kern.hz_max: 137438

[void]

Did adjusting it make any difference ?

I set it to maximum, rebooted but the vm wouldnt come up
fully lol, so made another one.

I'm going to try adjusting it in small increments to see if i can get the
virtio-net performance on freebsd guests to be as quick or nearly as quick as
linux ones in bhyve.

[Tomoaki AOKI]

On Tue, 15 Oct 2024 01:03:26 +0100

Post by void

Did adjusting it make any difference ?

I set it to maximum, rebooted but the vm wouldnt come up
fully lol, so made another one.

kern.hz dedines how frequently the forced task (process) switch happens.
The fewer the value is, the faster the process runs (lower overhead)
with the cost of lower responsiveness.

OTOH, the higher kern.hz is, the more chance to responding for user
interaction, with the cost of total performance loss (higher overhead).

These are because how much instructions could be run on single tick
(process/context switching) is affected with this. And this also affects
kernel, not only userland. And of course, affected by IPC and clock
freq of CPU cores and memory (including caches) bandwidth, too.

This means, setting too high value causes kernel to not finishing
needed-finishing-in-1-tick operations. Maybe your vm kernel thread
would be in this state.

Post by void
I'm going to try adjusting it in small increments to see if i can get the
virtio-net performance on freebsd guests to be as quick or nearly as quick as
linux ones in bhyve.
--

This tunable is defined as

SYSCTL_INT(_kern, OID_AUTO, hz, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &hz, 0,
"Number of clock ticks per second");
SYSCTL_INT(_kern, OID_AUTO, hz_max, CTLFLAG_RD, SYSCTL_NULL_INT_PTR,
HZ_MAXIMUM, "Maximum hz value supported");
SYSCTL_INT(_kern, OID_AUTO, hz_min, CTLFLAG_RD, SYSCTL_NULL_INT_PTR,
HZ_MINIMUM, "Minimum hz value supported");

in sys/kern/subr_param.c with related definitions and includes.
In this file, HZ is dedaulted as 1000 if not defined in other place or
make command line. Initially variable hz is set to -1, fetch for
tunable kern.hz, if none,

hz = vm_guest > VM_GUEST_NO ? HZ_VM : HZ;

is applied.

IIRC, in ancient days, default kern_hz (HZ) was 100 and bumped to 1000
(current default) at some point. So trying around 10000 or 5000 and
if not satisfactory, try increasing or decreasing would be nice for
recent amd64 hardwares. (It depends, though.)

FYI, I'm configuring kern.hz=4096 in /boot/loader.conf. Lost where I've
seen, but recommended by somewhere.

[Tomoaki AOKI]

On Tue, 15 Oct 2024 11:25:39 +0100

Post by void

Post by Tomoaki AOKI
FYI, I'm configuring kern.hz=4096 in /boot/loader.conf. Lost where I've
seen, but recommended by somewhere.

all offloading at source, host and destination turned off, default kern.hz in vm
================================================================================
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.06 sec 4.46 GBytes 637 Mbits/sec 1326 sender
[ 5] 0.00-60.07 sec 4.45 GBytes 637 Mbits/sec receiver
same as above, but with kern.hz=8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.02 sec 4.45 GBytes 637 Mbits/sec 1148 sender
[ 5] 0.00-60.02 sec 4.45 GBytes 637 Mbits/sec receiver
as above, kern.hz=4096
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.07 sec 3.09 GBytes 441 Mbits/sec 1013 sender
[ 5] 0.00-60.07 sec 3.09 GBytes 441 Mbits/sec receiver
With all offloading turned off, not seeing incorrect checksums at the
destination now.
--

Maybe kern.hz=4096 would be too large (short time per tick) for your
VMs, as VMs basically need more time than bare metal for emulated
(virtualized) syscalls or some instructions by mature.

And possibly (as I don't use offloading on VirtualBox that is the only
VM I tried) interactions between host (emulated hardware) and guests
about offloading has some issues.

[void]

Post by Tomoaki AOKI
IIRC, in ancient days, default kern_hz (HZ) was 100 and bumped to 1000
(current default) at some point. So trying around 10000 or 5000 and
if not satisfactory, try increasing or decreasing would be nice for
recent amd64 hardwares. (It depends, though.)
FYI, I'm configuring kern.hz=4096 in /boot/loader.conf. Lost where I've
seen, but recommended by somewhere.

The test vm here defaulted to 100. I got best throughput by turning it down to
8. Not by a lot though. I've commented out the hz value for now and rebooted:

% sysctl -a | ug hz
14: kern.clockrate: { hz = 100, tick = 10000, profhz = 8128, stathz = 127 }
191: kern.hz_min: 8
192: kern.hz_max: 137438
193: kern.hz: 100
6321: vfs.deferred_unmount.retry_delay_hz: 100
7299: debug.psm.hz: 20
8002: hw.atkbd.hz: 0

this is on -current built a few hrs earlier today

I'll try with hz as 0 or -1. I wish I could remember the linux article, as they
have a setting *something* like this and it's called something different.

[Dag-Erling Smørgrav]

Post by Dag-Erling Smørgrav

Post by void
I dimly remember a HZ setting, maybe kernel config option?
From years ago. I can't find it in recent /sys/amd64/conf/*

Look in sys/conf/NOTES.

not present in n271832-04262ed78d23 (-current)

https://cgit.freebsd.org/src/tree/sys/conf/NOTES#n1283

DES

[Stefan Bethke]

Post by void
I dimly remember a HZ setting, maybe kernel config option?
From years ago. I can't find it in recent /sys/amd64/conf/*
Does this functionality still exist or has it moved elsewhere?
Or did it ever exist?
I am seeing slow network throughput speeds in virtio-net relative to linux vms
on the same host and thought maybe it had something to do with polling too fast.

But it's not the buggy acceleration that virtio-net tries? I've turned off all offloading on my interfaces, as I was getting strange connectivity problems when it was enabled. Disabled is the default for OPNsense and many other appliances. See for example https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059.


Stefan

[void]

Post by void
I'll next try setting kern.hz to -1

setting it to -1 returns a kern.hz value of 100 on reboot
setting it to 0 returns the kern.hz_min value of 8
setting it to 1 and the kern.hz_min value to 1 still returns 8.

kern.hz=8 and kern.hz=100 gave the same numbers on bandwidth tests.

[Tomoaki AOKI]

On Tue, 15 Oct 2024 11:20:49 +0100

Post by void

Post by void
I'll next try setting kern.hz to -1

setting it to -1 returns a kern.hz value of 100 on reboot
setting it to 0 returns the kern.hz_min value of 8
setting it to 1 and the kern.hz_min value to 1 still returns 8.

-1 means "choose default".
Behaviors for 0 and 1 would be because HZ_MINIMUM is defined as 8 in
sys/sys/time.h, causing to clip to 8.

Post by void
kern.hz=8 and kern.hz=100 gave the same numbers on bandwidth tests.
--

Maybe kern.hz=100 gives enough time (per tick) and kernel returns near
the time even for kern.hz=8, disposing remaining time of the tick.

# There was an abandoned review D40045 to reuse the disposed remaining
# (fragmented) ticks. [1]


And forgot tomention in my previous post. kern.hz for VMs (recognized
as running on VM by FreeBSD kernel) is 100, as the default is HZ_VM.

HZ and HZ_VM is defined as 1000 and 100 respectively, if not defined
elsewhere. And if HZ_VM alone is not defined elsewhere, it is
defined as HZ.

These are because you see default kern.hz=100 on your VMs.

[1] https://reviews.freebsd.org/D40045

[void]

Post by Stefan Bethke
But it's not the buggy acceleration that virtio-net tries? I've turned off all offloading on my interfaces, as I was getting strange connectivity problems when it was enabled. Disabled is the default for OPNsense and many other appliances. See for example https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165059.

Thanks for this. I ran a tcpdump on the iperf3 'server' and not until I had
turned off all offloading completely - in vtnet0 in the vm, bge0 on the bhyve
server, and em0 on the iperf3 target, only then did incorrect checksums
disappear.

The bandwidth as measured with iperf3 seems to have decreased though.
But the effective bandwidth, I'd guess, has gone up because there's no more
reported checksum errors as measured at the target.

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.06 sec 4.46 GBytes 637 Mbits/sec 1326 sender
[ 5] 0.00-60.07 sec 4.45 GBytes 637 Mbits/sec receiver

I'll next try setting kern.hz to -1

[void]

Post by Tomoaki AOKI
FYI, I'm configuring kern.hz=4096 in /boot/loader.conf. Lost where I've
seen, but recommended by somewhere.

all offloading at source, host and destination turned off, default kern.hz in vm
================================================================================
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.06 sec 4.46 GBytes 637 Mbits/sec 1326 sender
[ 5] 0.00-60.07 sec 4.45 GBytes 637 Mbits/sec receiver

same as above, but with kern.hz=8
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.02 sec 4.45 GBytes 637 Mbits/sec 1148 sender
[ 5] 0.00-60.02 sec 4.45 GBytes 637 Mbits/sec receiver

as above, kern.hz=4096
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.07 sec 3.09 GBytes 441 Mbits/sec 1013 sender
[ 5] 0.00-60.07 sec 3.09 GBytes 441 Mbits/sec receiver

With all offloading turned off, not seeing incorrect checksums at the
destination now.