Very slow scp performance comparing to Linux

[Mikhail Zakharov]

Hi,

What is the drive back-end of your machines? I mean, what is seem to be
fast copied could still be cached in filesystem/storage buffers but not
actually written on the disk.

Best,
Mike

Hi,
When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same.
Here I was sending a 2GB file from sender to receiver using scp. FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu.
Sender Receiver throughput
Linux FreeBSD 70 MB/s
Linux Linux 550 MB/s
FreeBSD FreeBSD 70 MB/s
FreeBSD Linux 350 MB/s
FreeBSD localhost 70 MB/s
Linux localhost 550 MB/s
From theses test, it seems I can rule out the issue on NIC and its driver. Looks the FreeBSD kernel network stack is much slower than Linux on single stream TCP, or there are some problem with scp?
I also tried turning on following kernel parameters on FreeBSD kernel. But it makes no difference, neither do the other tcp cc algorithms such as htcp and newreno.
net.inet.tcp.soreceive_stream="1"
net.isr.maxthreads="-1"
net.isr.bindthreads="1"
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=419430
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=209715
kern.ipc.maxsockbuf=16777216
Any ideas?
Thanks,
Wei

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Guido Falsi]

Hi Mike,
All VMs are with same type and they are in Azure. If the copy on Linux is being cached on the RX side, so is FreeBSD?

You could perform the test using files in memory filesystems (tmpfs or
the like). This would factor out disk performance whatever the backend.

[Mateusz Guzik]

-----Original Message-----
Sent: Monday, August 28, 2023 6:49 PM
Subject: Re: Very slow scp performance comparing to Linux

Hi Mike,
All VMs are with same type and they are in Azure. If the copy on Linux
is

being cached on the RX side, so is FreeBSD?
You could perform the test using files in memory filesystems (tmpfs or
the
like). This would factor out disk performance whatever the backend.

Thanks, Guido. I tried tmpfs on the RX side with both FreeBSD and Linux VMs.
The throughput on FreeBSD went up significantly from 50 MB/s to 630 MB/s
with
NIC interface. Linux went up modestly form 550 MB/s to 660 MB/s.
So, looks in the non-tmpfs case, the Linux ext4 filesystem does cache large
amounts
of data in memory, much larger than FreeBSD ufs.
Many thanks, Mike for bringing this up and Guido, for the suggestion to try
tmpfs.

There is probably something funky going on here.

Here is a hack which will collect basic profiling info, works in a vm:

dtrace -w -n 'profile:::profile-4999 { @[sym(arg0)] = count(); }
tick-10s { system("clear"); trunc(@, 40); printa("%40a %@16d\n", @);
clear(@); }'

so start scp, start dtrace and see what happens

[Tomek CEDRO]

Post by Wojciech Puchar
linux filesystem caching is just stupid, optimized to show up good in
benchmarks.
It just accepts writes as long as there is available memory, without
writing anything to disk, then if it cannot do it anymore or some time
passed, schedules huge amount of writes, often stalling whole system.
This is how it was almost 20 years ago when i was still using linux, seems
like didn't change much.
Just think how much data loss and inconsistency would occur if there will
be crash or power loss after "writing" 100000 files of 10GB which can
easily be done in short time as linux just doesn't block writing process
at all and fill memory.
Since i started using FreeBSD UFS, many machines, lighter of heavier
workload, always mixed workload, i never ever have more that a few files
loss on power outage.

Exactly! I was recently surprised when my friend using Linux copied around
4GB file to my pendrive, it showed over 200MB/s and was rapid fast but then
we had to wait for background operation to complete for over an hour with
absolutely no information on progress/completion. This is why I prefer
FreeBSD :-)

--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

[Wojciech Puchar]

Post by Wojciech Puchar
Since i started using FreeBSD UFS, many machines, lighter of heavier
workload, always mixed workload, i never ever have more that a few files
loss on power outage.
Exactly! I was recently surprised when my friend using Linux copied around 4GB file to my pendrive, it showed over 200MB/s and was
rapid fast but then we had to wait for background operation to complete for over an hour with absolutely no information on

More funny it could stall the whole system or at least good part of it.
Pendrive is extreme example but it still exist for any media.

Now think what it would be if you run large multiservice server running
hundreds of different services and thousands of processes and you get
power failure.

Only linux make it possible :)


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Wei Hu <weh_at_microsoft.com> wrote on

When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same.
Here I was sending a 2GB file from sender to receiver using scp. FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu.
Sender Receiver throughput
Linux FreeBSD 70 MB/s
Linux Linux 550 MB/s
FreeBSD FreeBSD 70 MB/s
FreeBSD Linux 350 MB/s
FreeBSD localhost 70 MB/s
Linux localhost 550 MB/s
From theses test, it seems I can rule out the issue on NIC and its driver. Looks the FreeBSD kernel network stack is much slower than Linux on single stream TCP, or there are some problem with scp?
I also tried turning on following kernel parameters on FreeBSD kernel. But it makes no difference, neither do the other tcp cc algorithms such as htcp and newreno.
net.inet.tcp.soreceive_stream="1"
net.isr.maxthreads="-1"
net.isr.bindthreads="1"
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=419430
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=209715
kern.ipc.maxsockbuf=16777216
Any ideas?

You do not give explicit commands to try. Nor do you specify your
hardware context that is involved, just that HyperV is involved.

So, on a HoneyComb (16 cortex-A72's) with Optane boot media in
its PCIe slot I, no HyperV or VM involved, tried:

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 120.2MB/s 00:42

It is not a high performance system. 64 GiBytes of RAM.

So instead trying a ThreadRipper 1950X that also has Optane in a
CPIe slot for its boot media, no HyperV or VM involved,

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 299.7MB/s 00:17

(These systems do not run with any tmpfs areas, not even /tmp . So
I'm not providing that kind of example, at least for now.)

128 GiBytes of RAM.

Both systems are ZFS based but with a simple single partition.
(Used for bectl BE not for other types of reasons to use ZFS.
I could boot UFS variants of the boot media and test that
kind of context.)

So both show between your FreeBSD figure and the Linux figure.
I've no means of checking how reasonable the figures are relative
to your test context. I just know the results are better than
you report for localhost use.

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same.
Here I was sending a 2GB file from sender to receiver using scp. FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu.
Sender Receiver throughput
Linux FreeBSD 70 MB/s
Linux Linux 550 MB/s
FreeBSD FreeBSD 70 MB/s
FreeBSD Linux 350 MB/s
FreeBSD localhost 70 MB/s
Linux localhost 550 MB/s
From theses test, it seems I can rule out the issue on NIC and its driver. Looks the FreeBSD kernel network stack is much slower than Linux on single stream TCP, or there are some problem with scp?
I also tried turning on following kernel parameters on FreeBSD kernel. But it makes no difference, neither do the other tcp cc algorithms such as htcp and newreno.
net.inet.tcp.soreceive_stream="1"
net.isr.maxthreads="-1"
net.isr.bindthreads="1"
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=419430
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=209715
kern.ipc.maxsockbuf=16777216
Any ideas?

You do not give explicit commands to try. Nor do you specify your
hardware context that is involved, just that HyperV is involved.
So, on a HoneyComb (16 cortex-A72's) with Optane boot media in

I should have listed the non-debug build in use:

# uname -apKU

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 120.2MB/s 00:42
It is not a high performance system. 64 GiBytes of RAM.
So instead trying a ThreadRipper 1950X that also has Optane in a
CPIe slot for its boot media, no HyperV or VM involved,

I should have listed the non-debug build in use:

# uname -apKU
FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #116 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:20 PDT 2023 ***@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000

(Same source tree content.)

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 299.7MB/s 00:17
(These systems do not run with any tmpfs areas, not even /tmp . So
I'm not providing that kind of example, at least for now.)
128 GiBytes of RAM.
Both systems are ZFS based but with a simple single partition.
(Used for bectl BE not for other types of reasons to use ZFS.
I could boot UFS variants of the boot media and test that
kind of context.)
So both show between your FreeBSD figure and the Linux figure.
I've no means of checking how reasonable the figures are relative
to your test context. I just know the results are better than
you report for localhost use.

Adding a Windows Dev Kit 2023 booting via USB3 (but via a
U.2 adapter to Optane media), again ZFS, again no VM involved:

# uname -apKU
FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT 2023 ***@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 168.7MB/s 00:30


Note: the cortex-a72 and cortex-a78c/x1c builds were optimized via
-mcpu= use. The ThreadRipper build was not.


Note: I've not controlled for if the reads of the input *.img data
were gotten from memory caching of prior activity or not. I could
do so if you want: reboot before scp command.

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Wojciech Puchar]

1. ext4 caches a lot more than ufs?
2. there is a tcp performance gap in the network stack between FreeBSD and Ubuntu?
Would you also try run scp on ufs on your bare metal arm host? I am curious to now how different between ufs and zfs.

run on the same machine with FreeBSD directly. Compare to your VM host.

Personally i never run FreeBSD on VM except for tests. Exactly reverse
since bhyve exist.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Wei Hu <weh_at_microsoft.com> wrote on

Sorry for the top posting. But I don't want to make it look too messy. Here is the
Information that I have missed in my original email.
All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU).
14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 nodbg-n264692-59e706ffee52-dirty... /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64
6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s
Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 MB/s
Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs 48.8 Gb/s
Would these numbers suggest that
1. ext4 caches a lot more than ufs?
2. there is a tcp performance gap in the network stack between FreeBSD and Ubuntu?
Would you also try run scp on ufs on your bare metal arm host? I am curious to now how different between ufs and zfs.

For this round I'm rebooting between the unxz and the 1st scp.
So I'll also have zfs results again. I'll also do a 2nd scp
(no reboot) to see if it gets notably different results.

. . .

Well, I just got FreeBSD main [so: 15] running under
HyperV on the Windows Dev Kit 2023. So reporting for
there first. This was via an ssh session. The context
is ZFS. The VM file size is fixed, as is the RAM size.
6 cores (of 8) and 24576 MiBytes assigned (of 32
GiBytes) to the one FreeBSD instance. The VM file is
on the internal NVMe drive in the Windows 11 Pro file
system in the default place.

(I was having it copy the hardrive media to the VM file
when I started this process. Modern HyperV no longer
seems to support direct use of USB3 physical media. I
first had to produce a copy of the material on smaller
media so that a fixed VM file size from a copy to
create the VM file would fit in the NVMe's free space.)

# uname -apKU
FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT 2023 ***@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000

(The ZFS content is a copy of the USB3 interfaced
ZFS Optane media's content previously reported on.
So the installed system was built with -mcpu= based
optimization, as noted before.)

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 193.6MB/s 00:26

# rm ~/FreeBSD-14-TEST.img
# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.0MB/s 00:25


So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.

For reference:

# gpart show -pl
=> 40 468862055 da0 GPT (224G)
40 32728 - free - (16M)
32768 102400 da0p1 wdk23sCA78Cefi (50M)
135168 421703680 da0p2 wdk23sCA78Czfs (201G)
421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G)
468860928 1167 - free - (584K)

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x ONLINE -

(UFS would have notably more allocated and less free
for the same size partition.)



The below is be based on the HoneyComb (16 cortex-a72's)
since I've got the HyperV context going on the Windows
Dev Kit 2023 at the moment.


UFS first:

# uname -apKU
FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 ***@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 129.7MB/s 00:39

# rm ~/FreeBSD-14-TEST.img
# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 130.9MB/s 00:39


So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.

Note: This is via a U.2 Optane 960 GB media and an M.2 adapter
instead of being via a PCIe Optane 960 GB media in the PCIe
slot.


ZFS second:

# uname -apKU
FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 ***@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 121.1MB/s 00:42

# rm ~/FreeBSD-14-TEST.img
# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
(***@localhost) Password for ***@CA72-16Gp-ZFS:
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 124.6MB/s 00:41


So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.

Note: This is via a PCIe Optane 960 GB media in the
PCIe slot.


UFS was slightly faster then ZFS for the HoneyComb
context but there is the M.2 vs. PCIe difference
as well.


===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

[Adding USB3/U.2 Optane UFS Windows Dev Kit 2023 scp examples,
no VM's involved.]

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Sorry for the top posting. But I don't want to make it look too messy. Here is the
Information that I have missed in my original email.
All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU).
14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 nodbg-n264692-59e706ffee52-dirty... /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64
6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s
Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 MB/s
Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs 48.8 Gb/s
Would these numbers suggest that
1. ext4 caches a lot more than ufs?
2. there is a tcp performance gap in the network stack between FreeBSD and Ubuntu?
Would you also try run scp on ufs on your bare metal arm host? I am curious to now how different between ufs and zfs.

For this round I'm rebooting between the unxz and the 1st scp.
So I'll also have zfs results again. I'll also do a 2nd scp
(no reboot) to see if it gets notably different results.
. . .
Well, I just got FreeBSD main [so: 15] running under
HyperV on the Windows Dev Kit 2023. So reporting for
there first. This was via an ssh session. The context
is ZFS. The VM file size is fixed, as is the RAM size.
6 cores (of 8) and 24576 MiBytes assigned (of 32
GiBytes) to the one FreeBSD instance. The VM file is
on the internal NVMe drive in the Windows 11 Pro file
system in the default place.
(I was having it copy the hardrive media to the VM file
when I started this process. Modern HyperV no longer
seems to support direct use of USB3 physical media. I
first had to produce a copy of the material on smaller
media so that a fixed VM file size from a copy to
create the VM file would fit in the NVMe's free space.)
# uname -apKU
(The ZFS content is a copy of the USB3 interfaced
ZFS Optane media's content previously reported on.
So the installed system was built with -mcpu= based
optimization, as noted before.)
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 193.6MB/s 00:26
# rm ~/FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.0MB/s 00:25
So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.
# gpart show -pl
=> 40 468862055 da0 GPT (224G)
40 32728 - free - (16M)
32768 102400 da0p1 wdk23sCA78Cefi (50M)
135168 421703680 da0p2 wdk23sCA78Czfs (201G)
421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G)
468860928 1167 - free - (584K)
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x ONLINE -
(UFS would have notably more allocated and less free
for the same size partition.)
The below is be based on the HoneyComb (16 cortex-a72's)
since I've got the HyperV context going on the Windows
Dev Kit 2023 at the moment.
# uname -apKU
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 129.7MB/s 00:39
# rm ~/FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 130.9MB/s 00:39
So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.
Note: This is via a U.2 Optane 960 GB media and an M.2 adapter
instead of being via a PCIe Optane 960 GB media in the PCIe
slot.
# uname -apKU
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 121.1MB/s 00:42
# rm ~/FreeBSD-14-TEST.img
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 124.6MB/s 00:41
So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.
Note: This is via a PCIe Optane 960 GB media in the
PCIe slot.
UFS was slightly faster then ZFS for the HoneyComb
context but there is the M.2 vs. PCIe difference
as well.

# uname -apKU
FreeBSD CA78C-WDK23-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT 2023 ***@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000

Again, a -mcpu= optimized build context for the FreeBSD in
operation.

(Still rebooting first. Then . . .)

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 199.3MB/s 00:25

# rm ~/FreeBSD-14-TEST.img
# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 204.9MB/s 00:24


So, faster than what you are reporting for the
Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU)
context.

The Windows Dev Kit 2023 figures are generally faster than the
HoneyComb figures.

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
Hyper-V UEFI reports for the disk created:

1. SCSI Disk 0,0
The boot loader did not load an operating system.

(It then reports the network adapter attempt found no
boot image, but that is expected.)

That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.

Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.

I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.

Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.

The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.

Back to $work.


===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .

The kernel that results predates 6.0:

$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Using my usual rule of rebooting before the first scp:

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11

$ rm FreeBSD-14-TEST.img
$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10

Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).

I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.

For reference:

$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /



An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

So using 1024 MiByte of RAM assigned in Hyper-V:

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12

$ rm FreeBSD-14-TEST.img
$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12

Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).


===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Millard

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .
$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10
Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).
I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

The wording was poor: "force it" here is just from the
Ubuntu viewpoint. I make no claim to know if Hyper-V
is actually writing the material out to media at the
time vs. later.

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12
Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .
$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10
Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).
I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

The wording was poor: "force it" here is just from the
Ubuntu viewpoint. I make no claim to know if Hyper-V
is actually writing the material out to media at the
time vs. later.

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12
Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
of assigned RAM: use of localhost:/dev/null

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img

$ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 492.9MB/s 00:10


The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS context):

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img

# scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img ***@localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.7MB/s 00:25


Note: At most one VM running at a time, never both in overlapping times.

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .
$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10
Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).
I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

The wording was poor: "force it" here is just from the
Ubuntu viewpoint. I make no claim to know if Hyper-V
is actually writing the material out to media at the
time vs. later.

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12
Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
of assigned RAM: use of localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 492.9MB/s 00:10
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.7MB/s 00:25
Note: At most one VM running at a time, never both in overlapping times.

Avoiding having a cipher involved and even localhost
involved: use dd . . .


FreeBSD examples for Windows Dev Kit 2023 Hyper-V context,
24576 MiByts of RAM assigned):

# dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1m status=progress
2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1k status=progress
5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=512 status=progress
5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec)


Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context,
only 1024 MiBytes of RAM assigned:

$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1M status=progress
4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s
$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1K status=progress
4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s
***@ubwdk23s:~$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=512 status=progress
4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s


===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Saad]

All
Why not take scp out of the picture and try iperf? Why , we could be looking at rss by default in Linux .

---

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .
$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10
Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).
I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

The wording was poor: "force it" here is just from the
Ubuntu viewpoint. I make no claim to know if Hyper-V
is actually writing the material out to media at the
time vs. later.

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12
Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
of assigned RAM: use of localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 492.9MB/s 00:10
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.7MB/s 00:25
Note: At most one VM running at a time, never both in overlapping times.

Avoiding having a cipher involved and even localhost
involved: use dd . . .
FreeBSD examples for Windows Dev Kit 2023 Hyper-V context,
# dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1m status=progress
2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1k status=progress
5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=512 status=progress
5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec)
Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context,
$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1M status=progress
4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s
$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1K status=progress
4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s
4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s
===
Mark Millard
marklmi at yahoo.com

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Post by Mark Saad
All
Why not take scp out of the picture and try iperf? Why , we could be looking at rss by default in Linux .

The explorations with ???@localhost:/dev/null and with dd suggest
non-networking issues are a significant contributor to the data
rate differences being observed on FreeBSD vs. Ubuntu 22.04.3
--including when no networking is involved at all.

I also did experiments with cipher selections that I've not
reported.

As I've no clue why the original note was about specifically
about scp performance, I've just been try to gather data that
might be of some use, even for other contexts than just scp
used over an actual network.

Also having iperf figures for just the network contribution
would be useful too --if the network had appropriate
characteristics for comparison to the original context. The
network that I'm using is limited to 1 Gbit/s and may not be
a good match for comparison to the original context. I've
just not gone that direction so far.

Post by Mark Saad
---

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard

Post by Mark Millard
Wei Hu <weh_at_microsoft.com> wrote on

Thanks for the update. Seems the numbers are the same on zfs and ufs. That's
good to know.
Yes, your numbers on ARM64 are better than mine on Intel. However, my original
intention was to find out why scp on Linux is performing much better than FreeBSD
under the same hardware env.
Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 on ext4
file system.

I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023
to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the
1. SCSI Disk 0,0
The boot loader did not load an operating system.
(It then reports the network adapter attempt found no
boot image, but that is expected.)
That leaves me wondering if Hyper-V Quick Create
established a VM file holding Intel/AMD material
despite the aarch64 context.
Establishing a Ubuntu more directly is not familiar and
will have to be a background activity and, so, likely
will not be timely. If I did any experiments outside
Hyper-V (native booting), they would be with slower
USB3 SSD media than I use for FreeBSD.
I did notice that Hyper-V Quick Create did not create
a fixed sized disk but a dynamic sized one. That is
different than what I did for FreeBSD.
Also, it was not obvious if you were after aarch64
Hyper-V testing vs. native-boot testing vs. both. So
I may have gone the wrong direction from the start.
It is possible that I'd find establishing a native-boot
easier and then be able to have a VM file created from
the media, more like what I did with FreeBSD.
The Ubuntu activity likely would not be analogous to
the FreeBSD builds having -mcpu= optimization used.
Back to $work.

I found a sequence of UI operations that worked for
installing Ubuntu server 22.04.3 into Hyper-V in
Windows 11 Pro on the Windows Dev Kit 2023 via
use of a downloaded *.iso .
$ uname -ap
Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 431.3MB/s 00:11
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 482.2MB/s 00:10
Definitely faster than the FreeBSD results that I reported
earlier, including faster than the ThreadRipper 1950X with
Optane in a PCIe slot (more like 300 MiBytes/sec).
I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard
disk under Hyper-V.
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 squashfs 4.0 0 100% /snap/core20/1977
loop1 squashfs 4.0 0 100% /snap/lxd/24326
loop2 squashfs 4.0 0 100% /snap/snapd/19459
sda ├─sda1 vfat FAT32 F7E9-1344 1G 1% /boot/efi
└─sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% /
An experiment would be to have a small amount if RAM relative
the file size. That would force it to actually write to media
for some part of the file copy.

The wording was poor: "force it" here is just from the
Ubuntu viewpoint. I make no claim to know if Hyper-V
is actually writing the material out to media at the
time vs. later.

Post by Mark Millard
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 407.5MB/s 00:12
$ rm FreeBSD-14-TEST.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 404.7MB/s 00:12
Still definitely faster than the FreeBSD results that I
reported earlier, including faster than the ThreadRipper
1950X with Optane in a PCIe slot (more like 300 MiBytes/sec).

One more variation in ubuntu under Hyper-V, still with 1024 MiBytes
of assigned RAM: use of localhost:/dev/null
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 492.9MB/s 00:10
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img
. . .
FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img 100% 5120MB 198.7MB/s 00:25
Note: At most one VM running at a time, never both in overlapping times.

Avoiding having a cipher involved and even localhost
involved: use dd . . .
FreeBSD examples for Windows Dev Kit 2023 Hyper-V context,
# dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1m status=progress
2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s
5120+0 records in
5120+0 records out
5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1k status=progress
5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec)
CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=512 status=progress
5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec)
Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context,
$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1M status=progress
4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s
5120+0 records in
5120+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s
$ dd if=FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img of=/dev/null bs=1K status=progress
4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s
5242880+0 records in
5242880+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s
4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s
10485760+0 records in
10485760+0 records out
5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

-----Original Message-----
Sent: Thursday, August 31, 2023 10:18 AM
Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null
shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and
bs=512]

Post by Mark Saad
All
Why not take scp out of the picture and try iperf? Why , we could be looking

at rss by default in Linux .

Actually I did the iperf3 test as well and posted results a couple days ago.
FreeBSD iperf3 to localhost, single stream: 30.9 Gb/s
Linux iperf3 to localhost, single stream: 48.8 Gb/s

I'll note that (48.8Gb/s) / (30.9Gb/s) is a much
smaller ratio than then either of the originally
reported Ubuntu/FreeBSD ratios:

(550 MB/s) / (70 MB/s)
or:
(350 MB/s) / (70 MB/s)

Suggesting one or more non-network issues are
contributing, especially vs. the localhost test
that had the 550/70 ratio.

Neither of them has any tcp retry.
Both VMs run on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU).
The FreeBSD VM is 16 vcpu, with 128 GB memory.
The Linux VM is 4 vcpu, with 32 GB memory.
Wei

. . .

===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomek CEDRO]

Yes, the network perf gap is not as big as I originally thought. Seems different
factors contribute to the slowness of scp on FreeBSD. Network stack is only one
of them.

Note that Netflix uses FreeBSD because of _better_ network performance
over Linux.. also better long term maintenance that comes from better
organization, self-compatibility, and more stable kernel API that does
not change every minor release ;-)

Try the OS on a bare metal hardware, compare the results, use what
fits you best. If the results are similar then find a problem in the
VM hypervisor.

If you think of a long term project and comfortable maintenance use
FreeBSD even if you need to improve the driver it will pay back quite
soon.

[Wojciech Puchar]

Post by Mark Saad

Post by Mark Saad
All
Why not take scp out of the picture and try iperf? Why , we could be looking

at rss by default in Linux .

Actually I did the iperf3 test as well and posted results a couple days ago.
FreeBSD iperf3 to localhost, single stream: 30.9 Gb/s
Linux iperf3 to localhost, single stream: 48.8 Gb/s

Yes linux have lower system call latency, and somewhat more optimized
networking for special cases (like localhost).

For multiple processes using network results will be different.
But probably still be somewhat slower than linux. Just not 40% slower i
think


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Wojciech Puchar]

tested on really today's low end laptop
CPU: Intel(R) Pentium(R) CPU N3700 @ 1.60GHz (1600.19-MHz K8-class CPU)

got 73MB/s from tmpfs to tmpfs, most of it was user time of ssh
got 69MB/s from SSD to SSD, same.

Test on server computer
CPU: Intel(R) Pentium(R) CPU N3700 @ 1.60GHz (1600.19-MHz K8-class CPU)

337MB/s tmpfs to tmpfs
317MB/s ssd to ssd

too mostly user time (ssh and sshd, encryption/decryption).

I don't see anything wrong.

What to check:

a) linux ssh may have encryption off somehome
b) you tested in on slow storage

explanation of b - linux doesn't write to disk as long as it doesn't run
of of RAM. FreeBSD writes to disk just as large block is ready, then write
hell lot of data to disk, often blocking everything.

So Linux looks great in stupid benchmarks. FreeBSD gives you good
all-round performance with many processes working.



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Wojciech Puchar]

The throughput on FreeBSD went up significantly from 50 MB/s to 630 MB/s with
NIC interface. Linux went up modestly form 550 MB/s to 660 MB/s.
So, looks in the non-tmpfs case, the Linux ext4 filesystem does cache large amounts
of data in memory, much larger than FreeBSD ufs.

linux filesystem caching is just stupid, optimized to show up good in
benchmarks.

It just accepts writes as long as there is available memory, without
writing anything to disk, then if it cannot do it anymore or some time
passed, schedules huge amount of writes, often stalling whole system.

This is how it was almost 20 years ago when i was still using linux, seems
like didn't change much.

Just think how much data loss and inconsistency would occur if there will
be crash or power loss after "writing" 100000 files of 10GB which can
easily be done in short time as linux just doesn't block writing process
at all and fill memory.

Since i started using FreeBSD UFS, many machines, lighter of heavier
workload, always mixed workload, i never ever have more that a few files
loss on power outage.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de