Rust: kernel vs user-space

[Stefan Esser]

There are lots of control-plane things that I'd love to see
written mostly in Lua,

It was remiss of me to not mention Lua given that it's already in the project.
Yet another language which could make life easier, more productive and more accessible in
user-land.
I'm not suggesting for an instant that any of these programs need rewriting, but one could
imagine that if commands like ifconfig, route, arp, ndp, ipfw (that is, programs which
take a lot of user input and do a lot of data manipulation but aren't super-critical on
the performance front) were written in a more accessible language, then it might attract
new developers without disenfranchising the core C developers.

Here is ldconfig in LUA, written more than 2 years ago, for example:

https://github.com/stesser/ldconfig/blob/main/ldconfig.lua



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Rein Fernhout (Levitating)]

As for user-space, I find the rationale for Rust as the
one-true-language-after-C far less
compelling as many CLIs and server programs can just as well be written
in more accessible
languages such as go or perl or java or...
Frankly I no longer write any CLI or server code in C even after
decades of doing so
because the trade-off between development costs and performance is far
less compelling in
user-space. If my once-a-week invocation of a command requires a bit
more memory and CPU
than one written in C, is that really important compared to how much
easier the command is
to maintain and enhance?

I actually prefer Rust for CLI programs these days.
The Clap argument parser allows you to parse command line arguments,
with generated help and about pages, and autocomplete for various
shells, which also supports autocompleting hostnames and the likes, all
just by defining a structure.[0]

Rust can also use libc directly or bind to other C libraries and has a
very strong standard library in general.

Otherwise I would personally reach for Ruby or Python.

[0]: https://docs.rs/clap/latest/clap/_derive/index.html


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomek CEDRO]

What is FreeBSD ?

FreeBSD is a Free and Open-Source Unix Operating Systems with a
coherent and self sufficient base (/usr/src -compiler-> /) that can
expand userland with various Open-Source applications (/usr/ports
-compiler-> /usr/local).

FreeBSD "base" is a well defined common starting point after
installation always the same among different installations and
platforms. It can be used to bootstrap OS over network (diskless
install), hard drive, USB, any many others. From here custom userland
can be loaded and used. But the starting point is well defined and
common everywhere. I call this self-compatibility.

I like this design and would not change it.

Look at Linux for instance - each distribution is different, even
update of the same version of distribution can differ in
functionality. Not to mention kernel update will most likely break
userland due to driver API changes. Most problems on FreeBSD that are
related to incompatible modules (i.e. vbox or drm) are caused because
those modules are designed for Linux and adapted here. I call this
self-incompatiblity.

The difference in design is better seen on a resulting mobile phones -
all iOS (BSD based) devices have the same base that can be extended
with userland applications, while all Android (Linux based) devices
are completely different (base applications, bundled applications,
icons, even keyboard layout, etc).

Another example - you have zombie apocalypse and some disks with
source code for various operating systems - which systems would you be
able to build and run offline from those backups?

Have a good day folks :-)

[Dag-Erling Smørgrav]

Post by Tomek CEDRO
The difference in design is better seen on a resulting mobile phones -
all iOS (BSD based) devices have the same base that can be extended
with userland applications, while all Android (Linux based) devices
are completely different (base applications, bundled applications,
icons, even keyboard layout, etc).

The percentage of BSD-derived code in iOS rounds to 0.

DES

[Jan Knepper]

Yeah… I second that question…

(Not asking for a friend… :-) )



ManiaC++
Jan Knepper

Hi,

Post by Stefan Esser

There are lots of control-plane things that I'd love to see
written mostly in Lua,

It was remiss of me to not mention Lua given that it's already in the project.
Yet another language which could make life easier, more productive and more accessible in
user-land.
I'm not suggesting for an instant that any of these programs need rewriting, but one could
imagine that if commands like ifconfig, route, arp, ndp, ipfw (that is, programs which
take a lot of user input and do a lot of data manipulation but aren't super-critical on
the performance front) were written in a more accessible language, then it might attract
new developers without disenfranchising the core C developers.

https://github.com/stesser/ldconfig/blob/main/ldconfig.lua

And this is better than C because ...?
(Asking for a friend :-)
--
Bob Bishop

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Stefan Esser]

Hi,

Post by Stefan Esser

There are lots of control-plane things that I'd love to see
written mostly in Lua,

It was remiss of me to not mention Lua given that it's already in the project.
Yet another language which could make life easier, more productive and more accessible in
user-land.
I'm not suggesting for an instant that any of these programs need rewriting, but one could
imagine that if commands like ifconfig, route, arp, ndp, ipfw (that is, programs which
take a lot of user input and do a lot of data manipulation but aren't super-critical on
the performance front) were written in a more accessible language, then it might attract
new developers without disenfranchising the core C developers.

https://github.com/stesser/ldconfig/blob/main/ldconfig.lua

And this is better than C because ...?
(Asking for a friend :-)

There had been a request for an architecture independent implementation
that works on all FreeBSD releases and architectures.

The LUA run-time has been available in FreeBSD for a long time, and it
is a stable platform suitable for the implementation of commands that
operate on text and might take advantage of features like associative
storage. It is a "safe" implementation since memory is automatically
managed, which reduces the risk of memory management errors (but we
have to trust the LUA interpreter to be free of such issues, instead).

I have not made this version available for review (but instead updated
the C version to work with hints files of either byte order). But this
code is much simpler than the C version it replaces, and it could easily
be extended to, e.g., prepare hints files in chroot environments or in
images prepared for an architecture with a (possibly) different byte order.

And it is not only independent of the byte order and portable over all
architectures, but also independent of the FreeBSD version.

The LUA version is much shorter and easier to understand (if you know
LUA), but one reason not to propose it for inclusion in FreeBSD is that
there are many more developers that know how to work on the C version
than on the LUA version.

David and Mark have mentioned LUA, and there is little LUA code in the
base system, currently (mostly the loader). I'd like to see LUA being
used for more functionality that is not time critical, and since the LUA
language has been very stable over time (other than many other interpreted
languages), it is unlikely that a LUA script will not work on a later
FreeBSD release (it only depends on a working flua and possibly some
LUA include files).

Anyway, this is off-topic for the discussion about rust, other than
that LUA is a language that is not affected by many bugs that are
typical for programs written in C.

Best regards, STefan


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Steffen Nurpmeso]

Konstantin Belousov wrote in
<***@kib.kiev.ua>:
|On Wed, Sep 04, 2024 at 06:42:37PM +0200, Stefan Esser wrote:
|> Am 04.09.24 um 17:21 schrieb Bob Bishop:
|>>> On 4 Sep 2024, at 15:37, Stefan Esser <***@FreeBSD.org> wrote:
|>>> Am 04.09.24 um 11:52 schrieb Mark Delany:
|>>>> On 04Sep24, David Chisnall apparently wrote:
|>>>>> There are lots of control-plane things that I'd love to see
|>>>>> written mostly in Lua,
|>>>> It was remiss of me to not mention Lua given that it's already \
|>>>> in the project.
...
|>>> Here is ldconfig in LUA, written more than 2 years ago, for example:
|>>>
|>>> https://github.com/stesser/ldconfig/blob/main/ldconfig.lua
...
|The ldconfig utility is critical to fully configured multiuser FreeBSD
|state. Right now the utility depends on rtld and libc (ignoring the
|whole stuff at and below kernel). So any problem in either of rtld or
|libc which prevent ldconfig from run are critical.
|
|If rewritten in LUA, the utility has the same dependencies on rtld and
|libc, plus LUA runtime. Which, by itself, bring in
|/usr/libexec/flua:
| libm.so.5 => /lib/libm.so.5 (0x8010b1000)
| libedit.so.8 => /lib/libedit.so.8 (0x8010ed000)
| libprivateucl.so.1 => /usr/lib/libprivateucl.so.1 (0x801129000)
| libc.so.7 => /lib/libc.so.7 (0x80114f000)
| libtinfow.so.9 => /lib/libtinfow.so.9 (0x801552000)
|i.e. libm, terminfo, ucl (?), libedit. Then, flua runtime lives in /usr,
|while ldconfig is reasonably located in /sbin.
|
|There are other considerations than just the language accessibility when
|we consider the whole system.

Compared to monsters like clang/llvm FreeBSD *could* very well
include two lua binaries in maybe half a megabyte.
For Linux lua includes two make targets, as i learned just now.

FreeBSD NetBSD OpenBSD freebsd:
$(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE -I/usr/include/edit" SYSLIBS="-Wl,-E -ledit" CC="cc"
...
Linux linux: linux-noreadline

linux-noreadline:
$(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl"

linux-readline:
$(MAKE) $(ALL) SYSCFLAGS="-DLUA_USE_LINUX -DLUA_USE_READLINE" SYSLIBS="-Wl,-E -ldl -lreadline"

--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[David Chisnall]

Post by Stefan Esser
The LUA version is much shorter and easier to understand (if you know
LUA)

I am far from a Lua expert. I started using it quite recently because the best choice for a build system for the RTOS I maintain was written in Lua. I find it overly verbose at times, but, in spite of this, I rarely have problems reading other people’s Lua code.

It took me about an hour to go from never having written any Lua to writing some Lua code that actually worked (and that we still use). I see that as a huge advantage. There’s a lot of C code in the base system that took me ages to understand. Some, such as rtld, because it’s intrinsically complicated (though the fact that the authors are allergic to documentation doesn’t help: there are subtle phase ordering things there that should not have been committed without comments explaining what invariants future changes must preserve) but a lot of command-line tools are doing things that are fairly simple but the code is significantly complicated by the fact that C lacks abstractions for them and so implementation detail of data structures is interleaved with algorithms.

David

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jan Knepper]

Post by David Chisnall

Post by Stefan Esser
The LUA version is much shorter and easier to understand (if you know
LUA)

It took me about an hour to go from never having written any Lua to writing some Lua code that actually worked (and that we still use).

Honestly...
With the decades of coding experience I have, it seems that way with
most, if not all, other programming languages I learn...
I guess such is the benefit of having 'experience'... :-)



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Bertrand Petit]

Post by David Chisnall
It took me about an hour to go from never having written any Lua to
writing some Lua code that actually worked (and that we still use).

I made the same observation here, I must add that having a
funcional programming background helps greatly. The conciseness of the
reference manual is lovely even if I think it is badly organised.

I regret the "any variable name is assumed to be global unless
explicitly declared as a local" concept which preclude users from
writing large programs easily. 1-indexed strings is... peculiar by
modern times. I see the language libraries as very limited and
unsuitable for system programming---I was frustrated to not find map
nor reduce. If Lua is to be integrated into base for public
consumption we should beforehand write an extensive library of tool
functions that would include at least strings and tables
manipulations, functional abstractions, a sanctioned object system and
a complete set of system interfaces. By extensive I mean
Python-scale. That is a large project, do we have sufficient
ressources to conduct it from design to documentation?
--
%!PS -- Bertrand Petit
/D{def}def/E{exch}D/G{get}D/I{2 div}D/U{dup}D/L{roll}D/Y{setgray}D/N{newpath}D
/O{N 0 0 moveto}D/P{pop}D/T{translate}D currentpagedevice/PageSize G U 0 G/w E
D 1 G /h E D w I h I T 0 Y 1 setlinewidth 0 1 2 { P 120 rotate 2 4 w U mul h U
mul add sqrt I 50 add {N 50 0 3 2 L 0 360 arc stroke}for}for/s{O true charpath
pathbbox exch 4 -1 L E sub I 3 1 L sub I} D /l(bp)D 0.94 Y /Helvetica findfont
22 scalefont setfont l s P(x)s exch P T O l show showpage


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Kyle Evans]

Post by Bertrand Petit

Post by David Chisnall
It took me about an hour to go from never having written any Lua to
writing some Lua code that actually worked (and that we still use).

I made the same observation here, I must add that having a
funcional programming background helps greatly. The conciseness of the
reference manual is lovely even if I think it is badly organised.
I regret the "any variable name is assumed to be global unless
explicitly declared as a local" concept which preclude users from
writing large programs easily. 1-indexed strings is... peculiar by
modern times. I see the language libraries as very limited and
unsuitable for system programming---I was frustrated to not find map
nor reduce. If Lua is to be integrated into base for public
consumption we should [...]

We will not be exposing flua for general public consumption, but
anything in base is welcome to use it and we're happy to expand our
collection of modules as needed.

Thanks,

Kyle Evans


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Bridger Dyson-Smith]

Hi Bertrand

On Thu, Sep 5, 2024 at 1:19 PM Bertrand Petit <

Post by Bertrand Petit

Post by David Chisnall
It took me about an hour to go from never having written any Lua to
writing some Lua code that actually worked (and that we still use).

I made the same observation here, I must add that having a
funcional programming background helps greatly. The conciseness of the
reference manual is lovely even if I think it is badly organised.
I regret the "any variable name is assumed to be global unless
explicitly declared as a local" concept which preclude users from
writing large programs easily. 1-indexed strings is... peculiar by
modern times. I see the language libraries as very limited and
unsuitable for system programming---I was frustrated to not find map
nor reduce. If Lua is to be integrated into base for public
consumption we should beforehand write an extensive library of tool
functions that would include at least strings and tables
manipulations, functional abstractions, a sanctioned object system and
a complete set of system interfaces. By extensive I mean
Python-scale. That is a large project, do we have sufficient
ressources to conduct it from design to documentation?

Do you think incorporating a third-party Lua module would be acceptable?
E.g. https://github.com/lunarmodules/Penlight

I see map, reduce, string manipulation, and more.
--

Post by Bertrand Petit
%!PS -- Bertrand Petit
/D{def}def/E{exch}D/G{get}D/I{2
div}D/U{dup}D/L{roll}D/Y{setgray}D/N{newpath}D
/O{N 0 0 moveto}D/P{pop}D/T{translate}D currentpagedevice/PageSize G U 0 G/w E
D 1 G /h E D w I h I T 0 Y 1 setlinewidth 0 1 2 { P 120 rotate 2 4 w U mul h U
mul add sqrt I 50 add {N 50 0 3 2 L 0 360 arc stroke}for}for/s{O true charpath
pathbbox exch 4 -1 L E sub I 3 1 L sub I} D /l(bp)D 0.94 Y /Helvetica findfont
22 scalefont setfont l s P(x)s exch P T O l show showpage
Best,

Bridger

[Bertrand Petit]

Post by Bridger Dyson-Smith
Do you think incorporating a third-party Lua module would be acceptable?
E.g. https://github.com/lunarmodules/Penlight

That looks promising, it is modelled after Python batteries, that is a
good thing. When evaluating such libraries, after glancing at the range of
tackled problems and the interfaces proposed to solve them, because it is a
difficult problem, I always focus on date and time handling. It is not rosy,
see [1]. Right now I can't pronounce myself for or against the bulk of the
library, one must actually use it to forge an opinion, I can only say that the
documentation is not par to my expectations--the output of a doxygen-like tool
is never good enough IMHO.

The only Lua code I have is running in kernel as a set of ZFS channel
programs, an environment where one can't import libraries, this make testing
Penlight difficult for me. Testing must await for a medium size project not
requiring network access which I should write in Lua instead of Python.


[1] https://github.com/lunarmodules/Penlight/issues/285
--
%!PS -- Bertrand Petit
/D{def}def/E{exch}D/G{get}D/I{2 div}D/U{dup}D/L{roll}D/Y{setgray}D/N{newpath}D
/O{N 0 0 moveto}D/P{pop}D/T{translate}D currentpagedevice/PageSize G U 0 G/w E
D 1 G /h E D w I h I T 0 Y 1 setlinewidth 0 1 2 { P 120 rotate 2 4 w U mul h U
mul add sqrt I 50 add {N 50 0 3 2 L 0 360 arc stroke}for}for/s{O true charpath
pathbbox exch 4 -1 L E sub I 3 1 L sub I} D /l(bp)D 0.94 Y /Helvetica findfont
22 scalefont setfont l s P(x)s exch P T O l show showpage


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Konstantin Belousov]

Post by Stefan Esser

Hi,

Post by Stefan Esser

There are lots of control-plane things that I'd love to see
written mostly in Lua,

It was remiss of me to not mention Lua given that it's already in the project.
Yet another language which could make life easier, more productive and more accessible in
user-land.
I'm not suggesting for an instant that any of these programs need rewriting, but one could
imagine that if commands like ifconfig, route, arp, ndp, ipfw (that is, programs which
take a lot of user input and do a lot of data manipulation but aren't super-critical on
the performance front) were written in a more accessible language, then it might attract
new developers without disenfranchising the core C developers.

https://github.com/stesser/ldconfig/blob/main/ldconfig.lua

And this is better than C because ...?
(Asking for a friend :-)

There had been a request for an architecture independent implementation
that works on all FreeBSD releases and architectures.
The LUA run-time has been available in FreeBSD for a long time, and it
is a stable platform suitable for the implementation of commands that
operate on text and might take advantage of features like associative
storage. It is a "safe" implementation since memory is automatically
managed, which reduces the risk of memory management errors (but we
have to trust the LUA interpreter to be free of such issues, instead).
I have not made this version available for review (but instead updated
the C version to work with hints files of either byte order). But this
code is much simpler than the C version it replaces, and it could easily
be extended to, e.g., prepare hints files in chroot environments or in
images prepared for an architecture with a (possibly) different byte order.
And it is not only independent of the byte order and portable over all
architectures, but also independent of the FreeBSD version.
The LUA version is much shorter and easier to understand (if you know
LUA), but one reason not to propose it for inclusion in FreeBSD is that
there are many more developers that know how to work on the C version
than on the LUA version.

I do not think that the population of developers who knows LUA vs C
is significant for this example. But a different aspect is IMO much
more important.

The ldconfig utility is critical to fully configured multiuser FreeBSD
state. Right now the utility depends on rtld and libc (ignoring the
whole stuff at and below kernel). So any problem in either of rtld or
libc which prevent ldconfig from run are critical.

If rewritten in LUA, the utility has the same dependencies on rtld and
libc, plus LUA runtime. Which, by itself, bring in
/usr/libexec/flua:
libm.so.5 => /lib/libm.so.5 (0x8010b1000)
libedit.so.8 => /lib/libedit.so.8 (0x8010ed000)
libprivateucl.so.1 => /usr/lib/libprivateucl.so.1 (0x801129000)
libc.so.7 => /lib/libc.so.7 (0x80114f000)
libtinfow.so.9 => /lib/libtinfow.so.9 (0x801552000)
i.e. libm, terminfo, ucl (?), libedit. Then, flua runtime lives in /usr,
while ldconfig is reasonably located in /sbin.

There are other considerations than just the language accessibility when
we consider the whole system.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Cy Schubert]

In message <78BC157F-6E30-49C4-931D-***@digitaldaemon.com>, Jan
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be mirrored
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation as
provided by the D language were still 32-bit. I had opened an issue with
upstream regarding this. To this day they still haven't figured out how to
implement 64-bit inodes on newer FreeBSD systems while maintaining 32-bit
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).
--
Cheers,
Cy Schubert <***@cschubert.com>
FreeBSD UNIX: <***@FreeBSD.org> Web: https://FreeBSD.org
NTP: <***@nwtime.org> Web: https://nwtime.org

e^(i*pi)+1=0

ManiaC++
Jan Knepper

=20
=EF=BB=BFI hesitate to step into this discussion but is it worth making th=

e distinction between

Rust in the kernel and Rust in user-space?
=20
I can see the argument for introducing a "safer" language into the kernel a

=
nd there are

very few candidates available: perhaps only Rust, C++ and Zig. Clearly if t

=
hat step is to

be made, it probably should pick one language and run with it.
=20
That's one discussion.
=20
As for user-space, I find the rationale for Rust as the one-true-language-=

after-C far less

compelling as many CLIs and server programs can just as well be written in=

more accessible

languages such as go or perl or java or...
=20
Frankly I no longer write any CLI or server code in C even after decades o=

f doing so

because the trade-off between development costs and performance is far les=

s compelling in

user-space. If my once-a-week invocation of a command requires a bit more m

=
emory and CPU

than one written in C, is that really important compared to how much easie=

r the command is

to maintain and enhance?
=20
Point being, on the matter of introducing Rust to FreeBSD, I think the dis=

tinction between

kernel and user-space is worth keeping in mind as they are quite different=

problems.

=20
=20
Mark.
=20

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Konstantin Belousov]

Post by Jan Knepper
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be mirrored
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation as
provided by the D language were still 32-bit. I had opened an issue with
upstream regarding this. To this day they still haven't figured out how to
implement 64-bit inodes on newer FreeBSD systems while maintaining 32-bit
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).

Rust is same. It still uses pre-ino64 bindings for both stdlib and libc.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jacques Fourie]

Post by Cy Schubert
Jan

Post by Jan Knepper
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be

mirrored

Post by Jan Knepper
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation as
provided by the D language were still 32-bit. I had opened an issue with
upstream regarding this. To this day they still haven't figured out how

to

Post by Jan Knepper
implement 64-bit inodes on newer FreeBSD systems while maintaining

32-bit

Post by Jan Knepper
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).

Rust is same. It still uses pre-ino64 bindings for both stdlib and libc.

Looking at the Rust libc bindings I see the following:
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd11/mod.rs#L8
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd12/mod.rs#L5

Seems to have changed to 64 bit for FreeBSD 12 and up?

[Alan Somers]

Post by Jacques Fourie

Post by Cy Schubert
Jan

Post by Jan Knepper
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be

mirrored

Post by Jan Knepper
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation as
provided by the D language were still 32-bit. I had opened an issue with
upstream regarding this. To this day they still haven't figured out how

to

Post by Jan Knepper
implement 64-bit inodes on newer FreeBSD systems while maintaining

32-bit

Post by Jan Knepper
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).

Rust is same. It still uses pre-ino64 bindings for both stdlib and libc.

https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd11/mod.rs#L8
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd12/mod.rs#L5
Seems to have changed to 64 bit for FreeBSD 12 and up?

Rust libc seems to make some strange things to follow FreeBSD ABI
evolution, which is not done e.g. for glibc. But anyway, the relevant place
to look seems to be a comment and decision code at
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/build.rs#L44
By default they seems to use FreeBSD 11 ABI from freebsd11 module still.
I do not know what is CARGO_FEATURE_RUSTC_DEP_OF_STD.

The short answer is that Rust binaries still assume a FreeBSD 11 ABI.

The slightly longer answer is that while definitions are in place for
FreeBSD 12, 13, and 14, there's actually no way to enable them outside
of libc's CI system. CARGO_FEATURE_RUSTC_DEP_OF_STD is a variable set
while building the standard library. It allows the standard library
to use a different FreeBSD ABI version than libc.

And here is the PR that will finally raise the ABI to FreeBSD 12.
It's not getting much attention, unfortunately, because none of either
libc nor rustc's maintainers are FreeBSD developers, and they're
afraid to make changes like this.

https://github.com/rust-lang/libc/pull/2406


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jacques Fourie]

Post by Jacques Fourie

Post by Jacques Fourie

Post by Cy Schubert
Jan

Post by Jan Knepper
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be

mirrored

Post by Jan Knepper
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation

as

Post by Jacques Fourie

Post by Cy Schubert

Post by Jan Knepper
provided by the D language were still 32-bit. I had opened an issue

with

Post by Jacques Fourie

Post by Cy Schubert

Post by Jan Knepper
upstream regarding this. To this day they still haven't figured out

how

Post by Jacques Fourie

Post by Cy Schubert
to

Post by Jan Knepper
implement 64-bit inodes on newer FreeBSD systems while maintaining

32-bit

Post by Jan Knepper
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).

Rust is same. It still uses pre-ino64 bindings for both stdlib and

libc.
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd11/mod.rs#L8
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/src/unix/bsd/freebsdlike/freebsd/freebsd12/mod.rs#L5

Post by Jacques Fourie
Seems to have changed to 64 bit for FreeBSD 12 and up?

Rust libc seems to make some strange things to follow FreeBSD ABI
evolution, which is not done e.g. for glibc. But anyway, the relevant
place
to look seems to be a comment and decision code at
https://github.com/rust-lang/libc/blob/72c40004a3568849055c0bab5c92c9975b4eb132/build.rs#L44
By default they seems to use FreeBSD 11 ABI from freebsd11 module still.
I do not know what is CARGO_FEATURE_RUSTC_DEP_OF_STD.
CARGO_FEATURE_RUSTC_DEP_OF_STD is an environment variable set by cargo

when libc is built with the rustc-dep-of-std feature, as is done when
building the std library. What this means is that the std library uses a
libc ABI that is backwards compatible with FreeBSD 12. When using a
standalone libc from crates.io you will get libc bindings that are
backwards compatible with FreeBSD 11. Definitely not entirely clear to me.
I made a small sample app that prints the size of libc::ino_t. Building
this app without any environment variables defined results in a size of 4.
Building with `LIBC_CI=1` results in a size of 8, as does building with
`CARGO_FEATURE_RUSTC_DEP_OF_STD=1`. These tests were done on a host running
FreeBSD 15.

[Chris]

Post by Jan Knepper
Kneppe

D
www.dlang.org

The problem with D is data structure definitions need to also be mirrored
(duplicated) in D. For example, when 64-bit inodes were implemented D
failed to build and generate any code. The reason for this was
ufs/ufs/inode.h now defined 64-bit inodes while the D representation as
provided by the D language were still 32-bit. I had opened an issue with
upstream regarding this. To this day they still haven't figured out how to
implement 64-bit inodes on newer FreeBSD systems while maintaining 32-bit
inode backward compatibility on older FreeBSD systems (as FreeBSD
implemented this using ifunc).

Well, why about B?
https://en.wikipedia.org/wiki/B_(programming_language)

Sorry. I remembered using this *many* years ago, and couldn't resist
adding it to the list. :-)

--Chris

Post by Jan Knepper
--
Cheers,
e^(i*pi)+1=0

ManiaC++
Jan Knepper

=20
=EF=BB=BFI hesitate to step into this discussion but is it worth making th=

e distinction between

Rust in the kernel and Rust in user-space?
=20
I can see the argument for introducing a "safer" language into the kernel a

=
nd there are

very few candidates available: perhaps only Rust, C++ and Zig. Clearly if t

=
hat step is to

be made, it probably should pick one language and run with it.
=20
That's one discussion.
=20
As for user-space, I find the rationale for Rust as the one-true-language-=

after-C far less

compelling as many CLIs and server programs can just as well be written in=

more accessible

languages such as go or perl or java or...
=20
Frankly I no longer write any CLI or server code in C even after decades o=

f doing so

because the trade-off between development costs and performance is far les=

s compelling in

user-space. If my once-a-week invocation of a command requires a bit more m

=
emory and CPU

than one written in C, is that really important compared to how much easie=

r the command is

to maintain and enhance?
=20
Point being, on the matter of introducing Rust to FreeBSD, I think the dis=

tinction between

kernel and user-space is worth keeping in mind as they are quite different=

problems.

=20
=20
Mark.
=20

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Leidinger]

What is FreeBSD ?
-----------------
Forget Rust for the moment, I promise I will come back to it.
FreeBSD as a project was created almost entirely as protest against
the incompetent "UNIX-industry" as it existed around 1990.
One of the stupidities we reacted against, was the unbundling of
the C-compiler: If you bought a UNIX system, the C-compiler would
cost you extra, even through everybody knew that the vendor got it
as part of their source license from AT&T.
So from the very start, FreeBSD decided to deliver "A complete UNIX
system with full source" and the only concession was that, hard
disks costing what they did, you could choose to not install the
manual pages and the source code on systems which did not need them.

We are waaaay past that. The world is not only ftp and smtp servers
anymore.

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

[...]

But a FreeBSD system recompiling itself from source is even rarer.

In your world. And in the world of some other people. But there are a
lot of worlds where this is not true. I have systems which are updated
from src, and use only packages which are build locally.

And when it does, LLVM, source code we import verbatim from an
entirely different project, and which no sane person would call
"Related to FreeBSD", takes up more than three quarters of the
compile time.
That is objectively absurd.
The only reason we do that, is because we stil have that outdated
"FreeBSD is src" emotional hangup.

I don't think so. I'm sure everyone agrees that FreeBSD is not only src.
You already told it yourself. It's src+ports+pkg + other add-ons which
all live in the FreeBSD ecosystem.

We need to find a contemporary and useful answer to "What is FreeBSD?"
The only answer I can think of
------------------------------
"FreeBSD is ports (some of those ports contain the kernel and
userland)"

I propose "FreeBSD is packages" (and at the same time it isn't, but all
of the before mentioned stuff). How those packages are generated doesn't
matter, this is an implementation detail. "make buildworld" can create
packages. "make installworld" can install those packages. "make
distribpkg" can copy the basepackages to a directory structure similar
to what poudriere does which would allow to use "make whatever" in src
and poudriere to generate a package set for your site, your product, or
only yourself if you want to. It also would allow to continue using
finger memory, and have those which which to do so update the stuff via
the old way instead of using pkg (while having it installed via pkg in
the background).

As part of the migration, we yank LLVM out of the src.

Together with the possibility to use an external toolchain, this is
possible. I would welcome it to have a "freebsd-15-llvm-YYYYMMDDnn" port
which can be build on all supported releases and contains a compiler
which is able to compile FreeBSD-15 instead of llvm in src. Ideally this
tollchain port doesn't contain much local patches, so that I could grab
the source of it on OSX or Linux and use it to build FreeBSD (well... I
wouldn't, so far I've build FreeBSD only on FreeBSD, but we have
committers which use this, so we should keep that in mind when we create
such a port).

This external toolchain support should be generic (and I cross fingers
that Shawn has the time to come up with something sensible /
extensible). In that case the discussion about which language to use
doesn't matter anymore. It would allow up the possibility to have the
non-optional stuff managed in src, and optional stuff managed in ports.
It would allow to play around with different languages and to provide
replacements in different languages in parallel. This means that any
alternative is able to prove itself. Without the bikesheeding of
theoretical stuff, but by presenting evidence.

LLVM does not belong in src by any sane criteria, and any microscopic
benefits of "tight integration" can be delivered with a
"toolchain-llvm"
(meta-)port.
The installer
The kernel package(s)
The userland package(s)

An implementation detail which only matters once we have the possibility
to chose from packages...

"pkg upgrade" also upgrade kernel and userland packages - Welcome to
the century of the fruitbat.

I still think "make installworld" should be able to update my system
(and I should not care if this is done by using pkg or not, e.g. by
pointing pkg to the buildworld-generated packages repo in OBJDIR).

A more generic out of the box view than what PHK started follows now,
not directed at PHK directly but to all of us:

I do not think this is all what is needed. This proposal is surely a
step into the (IMO) right direction, but we should also discuss where we
stand with what we have already and where we should go with that.

flua is there, but there is no integration of what we have (recently
there started a discussion to add some freebsd modules to make freebsd
basesystem stuff accessible from flua, which seems to go into a (IMO)
good direction).

Similar for C++. We have some C++ code in the tree, but we have no guide
for it. Not only in terms of extensions to our style-guide, but also in
terms of what we want to see in terms of good practices. C++ is complex,
had a lot of changes since I first seen it in 1999. IMO we should use
the most recent C++ standard which our oldest supported release is able
to compile (to be reviewed every time a release is EoL resp. with every
major update of the external toolchain for the oldest release we
support). And we should provide some guidelines of what we want to see
in our C++ code (as a very basic example: use smart pointers instead of
raw pointers, or X instead of Y). Some of this may be common knowledge,
or best practices, but IMO we should have a document of what we want to
see. We don't have that for C either, but I think we should have that
for C too. We want to attract new people, and something like that would
make it more easy for new people to join. Similar for our kernel
interfaces. We have several of them, but we have no document which
explains which kind of interface to use for which kind of stuff. And we
have multiple ways to do something in the kernel, some of them old stuff
replaced by new stuff in some places but not all places, and a document
which talks about those parts (e.g. new code should use X instead of Y,
or if you touch it, please also do a migration to Y at the same time).

I bring this up here, because this is part of our code debt (and there
is surely other stuff which should be included too). And every
discussion about C vs C++ vs language of the day is implicitly a
discussion about code debt. All the arguments about we don't need more
than C are valid. At the same time all the arguments about the need for
<insert your favorite language> are valid too. In the end it's about
code debt, about cognitive load, code quality assurance and so on, not
about the language itself. While we gain experience, we all don't get
younger, may have not slept well last night, may have a hangover,
medical issues, have a broken heart, or be distracted. Anything which
makes it more easy to produce a good result is important, and as a
project we don't have the luxury to be able to say "we always did it
like this". We are proud to provide possibilities, but not policies on
our OS. We should be open minded enough because of that to provide the
possibility to let people use any language, not only in userland, but
also the kernel. If someone comes up with a subset of C++ to be used in
the kernel, why not? If this needs a bit of support in src, why not? The
same for rust or any other language. We need to draw a sane line
somewhere, sure, but we should not blindly say no to it. Someone wants
to prove that language X is a good addition to the kernel and it needs
2k lines (an arbitrary number I just picked out of thin air, don't nail
me down on this number) of changes to make it possible which can not be
done as a module and does not come with a too big cost, why not (mark it
experimental, make it an option, ...)? In the end any touring complete
language with enough low level possibilities is suitable to write an OS,
and if the language of day is suitable or not, can be seen by letting it
proof itself in this regard. If it isn't suitable, no important new
stuff will be produced and it will vanish. If it is suitable to an
extend never imagined, it will take over the world and old stuff is
replaced. Time will tell, but nothing bad will happen if we allow the
possibility to give it a try and make it even easy to give it a try.

Bye,
Alexander.

[Poul-Henning Kamp]

--------

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.

The only reason the Rust advocates need to bring this up is /precisely/
because that is not the case.

If it were, they would just have added ports.

Post by Alexander Leidinger
In your world. And in the world of some other people. But there are a
lot of worlds where this is not true. I have systems which are updated
from src, and use only packages which are build locally.

Beware of selection bias.

"Somebody who compiles from src" is almost the literal definition of "committer".

In terms of all the FreeBSD running hardware out there, not even
one percent of one percent of the machines compile from src.

(Hint: Consumer electronics and server farms running FreeBSD)

Poul-Henning
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Leidinger]

--------

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.
The only reason the Rust advocates need to bring this up is /precisely/
because that is not the case.
If it were, they would just have added ports.

So you are promoting a Linux-distro style model?

Post by Alexander Leidinger
In your world. And in the world of some other people. But there are a
lot of worlds where this is not true. I have systems which are updated
from src, and use only packages which are build locally.

Beware of selection bias.

This argument goes in both directions.

"Somebody who compiles from src" is almost the literal definition of "committer".

No.
- specific needs for a kernel which can not be satisfied with GENERIC
- specific needs / interest for optional stuff (maybe also as part of
hardening)
- specific needs / interest to exclude some stuff (maybe also as part
of hardening)
- ...
For nothing of this I need to be a committer, or even to be a
contributor.

In terms of all the FreeBSD running hardware out there, not even
one percent of one percent of the machines compile from src.
(Hint: Consumer electronics and server farms running FreeBSD)

Netflix/Sony or other vendors providing a product have their own way of
handling this... I don't think we should include those in this part of
the argument.

And server farms may exactly be the case of compiling from source (once,
and distributing this via their own freebsd-update or shared obj, or
whatever mechanism, with the reason being local patches or specific
kernel builds or whatever). Surely not all, but some of them. Enough of
them to be relevant to consider.

Yes, we surely have much more people using our binary-only possibilities
than it was the case 20 years ago when I joined FreeBSD, but there are
enough people out there that we can not neglect an update from src. And
I was not talking about each _machine_ compiling their own stuff, but
there are surely more _sites_ which compile their own stuff than one
percent of one percent.

Bye,
Alexander.

[Rick Macklem]

Post by Tomek CEDRO
On Mon, Sep 9, 2024 at 11:08 PM Alexander Leidinger

Post by Alexander Leidinger

--------

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.
The only reason the Rust advocates need to bring this up is /precisely/
because that is not the case.
If it were, they would just have added ports.

So you are promoting a Linux-distro style model?

Although I have no idea how many build from sources, I do think being
able to do so is one of the best features of FreeBSD.
I would never want to try and build a Linux distro from sources.
I just built a Linux kernel and it took about 24hrs on the old hardware I have.
(Don't worry. I've only gone over to the "dark side" temporarily;-)

As for build times, I already find "make buildworld" takes way too long
for me, so adding another ginormous compiler won't make much difference.
(I depend on the universe machines, if/when I need to do a full buildworld.)
I suspect most that do builds from sources will find a machine that can do it
(or just let in run for days while they do other things).

I'll stay out of the language debate, I know nothing about Rust. I only hope.
if you do invest resources in it, that it remains useful for some time.
(I remember when CS grads of another Ontario U. would come out with
programming skills in Euclid and Turing. Then they would discover no one
else on the planet used these languages.)

rick

Post by Tomek CEDRO
I am afraid this is the underlying message and I am happy to see a lot
of resistance here :-)
--
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Poul-Henning Kamp]

--------

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
So you are promoting a Linux-distro style model?

No.

But you already knew that.

Please try to be less inflamatory.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Leidinger]

Hi,

It was a serious question. As you say "no", I fail to understand what you
propose. Please explain it in more detail, our cognitive picture of what is
meant by your proposal does not seem to match. Or my picture of a Linux
distro is not similar to what your picture of a Linux distro looks like.

Bye,
Alexander.
--
Send from a mobile device, please forgive brevity and misspellings.
Am 10. September 2024 08:31:45 schrieb "Poul-Henning Kamp"

--------

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
So you are promoting a Linux-distro style model?

No.
But you already knew that.
Please try to be less inflamatory.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

[Poul-Henning Kamp]

--------

Post by Alexander Leidinger
It was a serious question. As you say "no", I fail to understand what you
propose. Please explain it in more detail, our cognitive picture of what is
meant by your proposal does not seem to match. Or my picture of a Linux
distro is not similar to what your picture of a Linux distro looks like.

Linux is only a kernel, and that finnish dude does not care a hoot
what executable PID=1 might or might not be or do.

FreeBSD is a kernel /and/ a filesystem full of programs, scripts
and other files, which cosplays as a complete, high quality and
usable, UNIX system[1], on which you can run the programs you
want/need/care for.

Poul-Henning

[1] Because somebody somewhere still defends the trademark.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Leidinger]

--------

Post by Alexander Leidinger
It was a serious question. As you say "no", I fail to understand what you
propose. Please explain it in more detail, our cognitive picture of what is
meant by your proposal does not seem to match. Or my picture of a Linux
distro is not similar to what your picture of a Linux distro looks like.

Linux is only a kernel, and that finnish dude does not care a hoot
what executable PID=1 might or might not be or do.

And a Linux _distro_ is a kernel /and/ a filesystem full of programs,
scripts and other files, which cosplays as a complete, high quality
(depending on who you ask) and usable, UNIX system[1], on which you can
run the programs you want/need/care for.

I might add, that a linux distro is also a set of packages which form
the above. And by telling FreeBSD is ports you gave me the impression,
that you want to say that we should move away from src is base and to
move to "a set of packages handled by the ports collection". I also
understand it to blur the lines between pkgbase and packages from ports
and make it possible to install parts of the basesystem from ports...
which to me looks like what I just described in this paragraph as a
linux distro.

FreeBSD is a kernel /and/ a filesystem full of programs, scripts
and other files, which cosplays as a complete, high quality and
usable, UNIX system[1], on which you can run the programs you
want/need/care for.
Poul-Henning
[1] Because somebody somewhere still defends the trademark.

Bye,
Alexander.

[void]

Post by Poul-Henning Kamp
Beware of selection bias.
"Somebody who compiles from src" is almost the > > literal definition of
"committer".

I build from src all the time and have never committed.

Post by Poul-Henning Kamp
In terms of all the FreeBSD running hardware out
there, not even
one percent of one percent of the machines
compile from src.

How have you measured this? I've never been asked.
Surely I'm not alone.

From this and other threads there seems to be a lot of "stands to reason" sentiment without any actual data...

Wouldn't it be better to make decisions based on data?

---


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Tomek CEDRO]

On Mon, Sep 9, 2024 at 11:08 PM Alexander Leidinger

Post by Alexander Leidinger

--------

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.
The only reason the Rust advocates need to bring this up is /precisely/
because that is not the case.
If it were, they would just have added ports.

So you are promoting a Linux-distro style model?

I am afraid this is the underlying message and I am happy to see a lot
of resistance here :-)

[David Chisnall]

Post by Rick Macklem
As for build times, I already find "make buildworld" takes way too long
for me, so adding another ginormous compiler won't make much difference.

As with so many other things, this would be addressable if someone had the time to do it. A single developer at Microsoft can edit the code and build their own custom Windows VM image on a fairly low-end laptop, in spite of the fact that most laptops don’t have enough disk space for a full git clone of the Windows source tree. This is possible because of a combination of two things:

- git-vfs lets you have a local view of a remote git repo and fetches files on demand as they’re accessed. Don’t run `grep` over your source tree!
- A build system that has cloud caching, so if you have changed one file you’ll rebuild everything that depends on that file but download the artefacts for everything else.

It’s a bit sad that building Windows typically takes a developer less time than building FreeBSD. The engineering effort involved in the second part is huge though. It would probably need the Foundation to pay someone to work on it for a year (I think this would be well worth the investment, but that’s another discussion). I’m not sure if bmake can be extended enough to support this (Buck2 looks to be the most promising of the new build systems designed around this model but it hasn’t yet had a stable release and comes with big ‘do not use this yet’ warnings). This would also need infrastructure to make sure each commit to head or a stable branch was built in a trusted environment and pushed to the caches (and that random developers’ builds did not).

David

[Lowell Gilbert]

Post by Alexander Leidinger

--------

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.
The only reason the Rust advocates need to bring this up is
/precisely/ because that is not the case.
If it were, they would just have added ports.

So you are promoting a Linux-distro style model?

As I read it, PHK was saying that this is why a "Linux-distro
style model" would not work for adding Rust kernel support to
FreeBSD at this time.

The "this" in my previous paragraph and in the quoted bits
isn't entirely clear to me, but the effort of maintaining Rust
bindings for kernel functionality seems to be a significant
part of the apparent problem.

Be well.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Kim Shrier]

Post by Poul-Henning Kamp

Post by Alexander Leidinger

The source tree became our citadel: "FreeBSD is src". If something
was not in src, it was not FreeBSD.

We are way past that too, FreeBSD is src+ports+docs(+community).

Nope.
The only reason the Rust advocates need to bring this up is /precisely/
because that is not the case.
If it were, they would just have added ports.

Post by Alexander Leidinger
In your world. And in the world of some other people. But there are a
lot of worlds where this is not true. I have systems which are updated
from src, and use only packages which are build locally.

Beware of selection bias.
"Somebody who compiles from src" is almost the literal definition of "committer".

WHAT ARE YOU SMOKING ???
AFAIK everybody who needs patches is oblidged to recompile from
source. And I don't get kerberos to work without my patches, I dont
get netgraph to work without my patches, I don't get ipfw to work
without my patches.
And AFAIK committers are people who write NEW software. I am not
interested in that, I only need to make the existing software
actually work as intended for my site.

I have stayed out of this conversation so far but I agree with Peter.
I compile all my kernels and ports from source. I make local patches
when needed. And while I have had some of my patshes accepted
into the FreeBSD source, I am not a committer.

I have dabbled with rust, several times and I don’t like it. It reminds
me too much of C++ and I have never liked that language. These
languages appear to introduce too much complexity and their
runtime libraries change too much over time. If I were to consider
introducing a new language into the FreeBSD source tree, it would
be V but I know that that language is still too early in its development
to seriously consider using it… yet. I have high hopes for it.

So consider this a NO vote on rust.

Kim Shrier
-------------------
C++ is an off-by-one error





--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Borja Marcos]

Post by Kim Shrier
I have stayed out of this conversation so far but I agree with Peter.
I compile all my kernels and ports from source. I make local patches
when needed. And while I have had some of my patshes accepted
into the FreeBSD source, I am not a committer.

One of the outstanding advantages of FreeBSD versus the avian alternative is
the ports system.

There are many ports with meaningful compile time options. Typical conversation
with an avian user: “Ok, to make this work you need to tweak this compile time
option and compile” — and the answer is usually “whaaat? I don®t see that in the
repo, can’t do”.

Post by Kim Shrier
I have dabbled with rust, several times and I don’t like it. It reminds
me too much of C++ and I have never liked that language. These
languages appear to introduce too much complexity and their
runtime libraries change too much over time.

I agree with you regarding the runtime changes. The C family has many problems,
it's known since the 80’s.

But an alternative must be stable enough to be included with the base system. And recent
languages change way too much. And some changes are more aesthetic than useful, which
in my opinion doesn’t justify breaking stuff.

Memory safety is critical, it’s been way too long with buffer overflows and
other nonsense that should not happen in a high level language, but developer sanity
is also mandatory. Languages evolve, but there’s no need to evolve them yearly.





Borja.

[Joe Schaefer]

Correct take. Rust is clearly not a mature ecosystem suitable to be an
Operating System platform, commercial or otherwise.

Yes DARPA is funding it, like it funds bioterrorism. It’s not the same
organization is was 30 years ago.

Post by Borja Marcos

Post by Kim Shrier
I have stayed out of this conversation so far but I agree with Peter.
I compile all my kernels and ports from source. I make local patches
when needed. And while I have had some of my patshes accepted
into the FreeBSD source, I am not a committer.

One of the outstanding advantages of FreeBSD versus the avian alternative is
the ports system.
There are many ports with meaningful compile time options. Typical conversation
with an avian user: “Ok, to make this work you need to tweak this compile time
option and compile” — and the answer is usually “whaaat? I don®t see that
in the
repo, can’t do”.

Post by Kim Shrier
I have dabbled with rust, several times and I don’t like it. It reminds
me too much of C++ and I have never liked that language. These
languages appear to introduce too much complexity and their
runtime libraries change too much over time.

I agree with you regarding the runtime changes. The C family has many problems,
it's known since the 80’s.
But an alternative must be stable enough to be included with the base system. And recent
languages change way too much. And some changes are more aesthetic than useful, which
in my opinion doesn’t justify breaking stuff.
Memory safety is critical, it’s been way too long with buffer overflows and
other nonsense that should not happen in a high level language, but developer sanity
is also mandatory. Languages evolve, but there’s no need to evolve them yearly.
Borja.

[void]

Hi,

A FreeBSD system with no installed ports is a rarity today.

I have 3 such systems. There may be more people running plain systems than you think.
There's no way to tell. Although i concede most systems will have ports installed.

But a FreeBSD system recompiling itself from source is even rarer.

? really? All my -stable and -current machines are recompiled from source.
Is this really that rare?

We need to find a contemporary and useful answer to "What is FreeBSD?"

I've always thought of it as "freebsd is an OS". [1]
I couldn't care less if the compiler comes from somewhere else ultimately,
as long as it works and has a BSD or similar permissive licence.
Everything ultimately in some shape or another comes from somewhere.
If it's in src/ then it's FreeBSD IMO.

The only answer I can think of
------------------------------
"FreeBSD is ports (some of those ports contain the kernel and userland)"

Oh I hope not... ports can sometimes be a nightmare.
The separation of OS and ports is a great thing to have and is
a freebsd 'selling point' IMO.
One can easily deinstall all ports and blat /usr/local/* then
reinstall, in order to recover a damaged system, for example.

As part of the migration, we yank LLVM out of the src.
LLVM does not belong in src by any sane criteria, and any microscopic
benefits of "tight integration" can be delivered with a "toolchain-llvm"
(meta-)port.
And yes, we have ports written in Rust, why do you ask?

I don't think the benefits of tight integration are microscopic,
at least not to me. I'd rather have kernel modules in src than in ports,
including the video and wifi stuff. ports get out of sync and breakage happens.
This seems less of an issue for src.

At present, we have to install pkg to even get a tool for downloading sources.
FreeBSD as OS is in that sense incomplete (again IMO).
Why isn't got or ***@tiny in base. We used to have svnlite.

[1] If freebsd goes entirely to pkgbase, would it then really be an OS or
just a distro? (eg 'linux is a kernel')

[Jamie Landeg-Jones]

Post by void
? really? All my -stable and -current machines are recompiled from source.
Is this really that rare?

Mine too (well, I only track stable at the moment, and am only talking about 8
machines)

I have too many local patches to easily do it any other way. As it is, i sync
src, patches are automatically applied, then make buildworld etc... is all
that's needed.

Similar story for ports vs packages - too many patches (and custom options)



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Cy Schubert]

In message <***@donotpassgo.dyslexicfish.net>,
Jamie La

Post by Jamie Landeg-Jones

Post by void
? really? All my -stable and -current machines are recompiled from source.
Is this really that rare?

Mine too (well, I only track stable at the moment, and am only talking about 8
machines)
I have too many local patches to easily do it any other way. As it is, i sync
src, patches are automatically applied, then make buildworld etc... is all
that's needed.
Similar story for ports vs packages - too many patches (and custom options)

Those of us who build from source and build ports, whether manually or
through our own poudriere, are the minority. Just visit the FreeBSD forums.
I attend OpenHack here. People who do use FreeBSD use freebsd-update and
binary packages. (I use freebsd-update and binary packages on some VMs at
$JOB, while maintaining my own network at home as any developer does.)

And that's a marketing feature of FreeBSD. Most users don't want he hassle
of building and installing an O/S.

And a co-worker has set up an EC2 instance (thanks cpercival@).

Out in the real world people use binary updates and binary packages. We
developers are an anomaly these days.

Just because a few of us build from source doesn't mean the rest of the
world does.
--
Cheers,
Cy Schubert <***@cschubert.com>
FreeBSD UNIX: <***@FreeBSD.org> Web: https://FreeBSD.org
NTP: <***@nwtime.org> Web: https://nwtime.org

e^(i*pi)+1=0





--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Anthony Pankov]

Hello, Cy.

You wrote:
<overquoting deleted>

Post by Cy Schubert
Those of us who build from source and build ports, whether manually or
through our own poudriere, are the minority.

I think a proportion of FreeBSD audience who build from source vs use
binary is directly correlated to years of experience.

If a recent Windows update prevent users for doing something and
there is a samba fix... But only for latest samba version which still not in
ports collection...

You have only one way: rebuild your production samba package with
applied patch. Things goes smooth if your production package is
locally builded, you use poudriere overlay with this patch to get the same samba
version with the same dependencies matching those already in production.

All other way are very painfull.

Binary updaters vs source builders are splitted by the point when they hit
such a things. Consequently, persistent user who prefer binary
update inevitably become a source builder.

In my humble opinion.

[Daniel Engberg]

  Those of us who build from source and build ports, whether
  manually or 
  
   through our own poudriere, are the minority. Just visit the
  FreeBSD forums.

 
 IIRC, the forums don't entertain issues raised by src builders, 
 
 only -releng. This is from a while ago though, I might be wrong
 about that now,
 
 am happy to be corrected.
 

  I attend OpenHack here. People who do use FreeBSD use
  freebsd-update and 
  
   binary packages. (I use freebsd-update and binary packages on
  some VMs at 
  
   $JOB, while maintaining my own network at home as any developer
  does.)

 
 I use freebsd-update on some VMs too. It has its place. But always
 poudriere
 
 for ports, as most of the VMs are internet facing, and when a vuln
 happens
 
 and is patched it's the fastest way to fix the situation, rather
 than waiting
 
 on the pkg builders.
 

  And that's a marketing feature of FreeBSD. Most users don't want
  he hassle 
  
   of building and installing an O/S.

 
 Have most users been asked?
 

  Out in the real world people use binary updates and binary
  packages. We 
  
   developers are an anomaly these days.

 
 I'd not consider myself a dev. That might be just me though. Is
 streamlining a
 
 kernel to have what you want and no more a 'dev' activity? Manually
 patching?
 

  Just because a few of us build from source doesn't mean the rest
  of the 
  
   world does.

 
 How would you know? Who has counted the numbers? I think maybe a
 poll on the
 
 main site might be enlightening. I mean, I agree src builders are
 probably in 
 
 a minority now, as freebsd-update is convenient in standard cases,
 
 but it's possibly a larger number than you think, who build from
 src.
 
 We'll never really know without counting.
 
 I really hope that when pkgsrc becomes dominant, that we're still
 able to 
 
 grab src in git and checkout whats required, and build from that.
 
 It's so very versatile.
 
 --

I would imagine that for larger installs it's something inbetween
where you build your own "set" of packages and base with custom
settings etc and then push the binaries.

I would also like to remind people that at least for ports far from
all ports have runtime detection of SIMD instructions which can cause
quite a bit of a difference in performance so setting CPUTYPE might
drastically improve performance. Canonical (Ubuntu) are looking into
providing different sets of packages depending on baseline so it's a
thing. https://www.phoronix.com/news/Ubuntu-x86-64-v3-Images-Azure

I also build from source btw =)

Best regards,

Daniel

[Tomek CEDRO]

Not really good idea to divide people in general, and here to
generalize about people who build from source or use binary packages.

Its good to have binary packages just to quickly setup or upgrade some
machines, verify integrity, save time, etc.

Its good to have src and ports to build from sources - test fixes,
create or extend driver, add new port or update existing one, etc - or
to read sources to understand stuff. Sometimes it is the only way
possible.

Binary releases are probably more convenient for desktop and cloud/vm
use, source builds are more common among custom setups and embedded
systems.

Another place where source build is the only way is Python Virtual
Environment - usually packages for FreeBSD are not provided at all and
need to be compiled on site.

I find it kinda funny that its several times faster to build whole
FreeBSD kernel and/or base than for instance modern web browser :D

I also use FreeBSD to develop various RTOS based firmwares :-)

Both uses are important in Open-Source.

What is the problem exactly? :D

ps/2: It good to see so many folks active here on mailing lists :-)

[Jan Knepper]

Post by Cy Schubert
Jamie La

Post by Jamie Landeg-Jones

Post by void
? really? All my -stable and -current machines are recompiled from source.
Is this really that rare?

Mine too (well, I only track stable at the moment, and am only talking about 8
machines)
I have too many local patches to easily do it any other way. As it is, i sync
src, patches are automatically applied, then make buildworld etc... is all
that's needed.
Similar story for ports vs packages - too many patches (and custom options)

Those of us who build from source and build ports, whether manually or
through our own poudriere, are the minority. Just visit the FreeBSD forums.
I attend OpenHack here. People who do use FreeBSD use freebsd-update and
binary packages. (I use freebsd-update and binary packages on some VMs at
$JOB, while maintaining my own network at home as any developer does.)
And that's a marketing feature of FreeBSD. Most users don't want he hassle
of building and installing an O/S.
Out in the real world people use binary updates and binary packages. We
developers are an anomaly these days.
Just because a few of us build from source doesn't mean the rest of the
world does.

Probably...

Used to compile on every instance of FreeBSD I run.

Still do on many, but have moved to using freebsd-update on some.

(Just learned Saturday that is is _not_ a good idea to try to update
multiple classic jails in parallel (thus at the same time) with
freebsd-update. <LOL>)



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jamie Landeg-Jones]

Post by Cy Schubert
Those of us who build from source and build ports, whether manually or
through our own poudriere, are the minority. Just visit the FreeBSD forums.
I attend OpenHack here. People who do use FreeBSD use freebsd-update and
binary packages. (I use freebsd-update and binary packages on some VMs at
$JOB, while maintaining my own network at home as any developer does.)
And that's a marketing feature of FreeBSD. Most users don't want he hassle
of building and installing an O/S.

Sure, I realise that. And I'm not knocking binary installs - indeed
installing from a 2.2.6-release CD was how I got into FreeBSD, and I stuck
to that method for a while.

In fact, at the time, I had an intranet site at ICL (where I worked at the
time) titled something like "Install your own unix operating system in just
40 minutes"), where I raved about how easy and quick it was!

Post by Cy Schubert
Out in the real world people use binary updates and binary packages. We
developers are an anomaly these days.
Just because a few of us build from source doesn't mean the rest of the
world does.

I know, but many of the comments here have implied just about no-one builds
from source. I just wanted to readdress that balance.

Hell, one comment said if I compile from source, I must be a committer, but
I've yet to find that commit bit :-)

I don't mind that there's a focus on binary distributions; I don't mind if
source building gets a bit trickier; I just hope source building remains a
viable route for those of us who aren't part of the FreeBSD organisation.
We may be a minority, but there are still a number of us here!

Cheers, Jamie


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Jamie Landeg-Jones]

Post by Daniel Engberg
I would also like to remind people that at least for ports far from
all ports have runtime detection of SIMD instructions which can cause
quite a bit of a difference in performance so setting CPUTYPE might
drastically improve performance.

Very much this! (I didn't mention it in earlier replies for danger of
being called a "ricer", but I only add -march=native, not any other
ricer flags!)

Cherrs, Jamei


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[void]

Post by Cy Schubert
Those of us who build from source and build ports, whether manually or
through our own poudriere, are the minority. Just visit the FreeBSD forums.

IIRC, the forums don't entertain issues raised by src builders,
only -releng. This is from a while ago though, I might be wrong about that now,
am happy to be corrected.

Post by Cy Schubert
I attend OpenHack here. People who do use FreeBSD use freebsd-update and
binary packages. (I use freebsd-update and binary packages on some VMs at
$JOB, while maintaining my own network at home as any developer does.)

I use freebsd-update on some VMs too. It has its place. But always poudriere
for ports, as most of the VMs are internet facing, and when a vuln happens
and is patched it's the fastest way to fix the situation, rather than waiting
on the pkg builders.

Post by Cy Schubert
And that's a marketing feature of FreeBSD. Most users don't want he hassle
of building and installing an O/S.

Have most users been asked?

Post by Cy Schubert
Out in the real world people use binary updates and binary packages. We
developers are an anomaly these days.

I'd not consider myself a dev. That might be just me though. Is streamlining a
kernel to have what you want and no more a 'dev' activity? Manually patching?

Post by Cy Schubert
Just because a few of us build from source doesn't mean the rest of the
world does.

How would you know? Who has counted the numbers? I think maybe a poll on the
main site might be enlightening. I mean, I agree src builders are probably in
a minority now, as freebsd-update is convenient in standard cases,
but it's possibly a larger number than you think, who build from src.

We'll never really know without counting.

I really hope that when pkgsrc becomes dominant, that we're still able to
grab src in git and checkout whats required, and build from that.
It's so very versatile.

[Eirik Øverby]

Post by Cy Schubert
Just because a few of us build from source doesn't mean the rest of the
world does.

I didn't think I'd reply on any of these threads, but here I go: I (or we, at work), sometimes build from source if we need a patch applied or a test run in some setting or other. We build our own ports/packages - but *not on the target system*. So for each machine we have that builds anything from source, we have a couple hundred physical and virtual ones that use binary updates: freebsd-update and pkg.

It is *absolutely essential* to promote pkgbase to first-class citizen. If for none of the reasons discussed in these threads, simply because a coherent approach to installing and updating our systems would make our lives so, so, so! much easier! Right now, we have *three* sources of binaries to get a single jail up and running and keep it maintained:
- downloading base.txz from somewhere to create the jail template
- freebsd-update to keep host and template updated
- pkg to install and maintain 3rd party packages

This, in turn, makes it necessary to manage three different fetch, build, caching and distribution mechanisms. Yes we could use pkgbase in-house, but we need to run "official builds" for the base OS ("compliance", yuck!).

As has been said elsewhere, and I'll spell it out: Absolutely *nothing* will be lost by moving to pkgbase and making it the default distribution, installation and updating mechanism. And absolutely nothing about that would prevent anyone from doing precisely what they're doing today, in exactly the same way.

Sorry for the wall of text. There's been a lot of them lately :)

/Eirik


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Chris]

Post by void
Hi,

A FreeBSD system with no installed ports is a rarity today.

I have 3 such systems. There may be more people running plain systems than you think.

...

Post by void
FreeBSD as OS is in that sense incomplete (again IMO).
[1] If freebsd goes entirely to pkgbase, would it then really be an OS or
just a distro? (eg 'linux is a kernel')

I'm just going to respond to this last bit...
It wouldn't be a "distro" in the sense of Linux, as Linux is only a kernel.
Whereas (Free)BSD is a complete system.

P.S. I know this reply comes late; but I'm catching up on email. :(

--Chris


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Olivier Certner]

Hello Poul-Henning,

But a FreeBSD system recompiling itself from source is even rarer.
(...)
Beware of selection bias.
"Somebody who compiles from src" is almost the literal definition of "committer".
In terms of all the FreeBSD running hardware out there, not even
one percent of one percent of the machines compile from src.

You're most probably right. At the same time, I'm having a hard time finding this angle of the number of machines compiling 'src' even remotely relevant to the debate.

As part of determining LLVM's position in FreeBSD (and, in fact, more), the crucial questions seem to me to be: For which uses do we need to compile from source and what is the importance (in the widest possible sense) of these uses to the project?

To state the obvious, committers and developers have to build from source, so it must be easy for them to do so, and must *remain* so. But there is much more.

Building customized version of FreeBSD is very useful to fulfill at least these needs:
- Usability/Performance: Stripped-down OSes for embedded platforms, and more recently VMs.
- Security: Reduce the attack surface by removing not used components.
- Appliances (for the various reasons above).
- Experiments: Experimental options, drivers, etc.

Being able to re-build releases in a controlled environment (with their associated specific version of the compiler) is useful for maintenance (bug fixing, debugging) and security (build reproducibility).

And compiling from source is likely also very valuable for attracting new developers. From an educational standpoint, it may foster adoption in the cursus of more universities, raising the exposure of students to FreeBSD and our chances to have them as developers later on.

These are already prominent reasons to make sure that building from source is not only feasible, but actually *easy* to do. And I'm sure this list is not even complete.

LLVM does not belong in src by any sane criteria, and any microscopic
benefits of "tight integration" can be delivered with a "toolchain-llvm"
(meta-)port.

The benefits of "tight integration" are by no means "microscopic" (easy to setup environment, reproducible builds, compiler bugs workaround, etc.). I wonder what makes you think otherwise.

However, I agree that tight integration per se doesn't necessarily require having LLVM/clang's code living in 'src'. It could be achieved through something like a port, although for now it's unclear to me if exactly a port as we have today would be feasible or desirable (more on this below).

The external toolchain infrastructure is an already existing step in this direction. If we chose to build on it, I think what would be missing is:
- The way to record a dependency to a precise version of the compiler(s) (plus custom patches) needed to build the sources.
- The ability to easily (a command away, or even automatically as part of the build) grab these compilers in binary form.
- Same, but to grab their code (and apply patches) and build them with some already installed compilers (the result may be used directly to build 'src' or as a boostrap).
But see more below.

We need to find a contemporary and useful answer to "What is FreeBSD?"
Delivering a single consistent userland with the kernel has stood
us well for three decades, and we should stick with that.

I'll add:
- A system that is easy to build and tweak in practice (for developers at the very least).
- A system that is self-sufficient, in the sense that, once installed on a machine, one can set it up to do whatever one would like without having to boot something else (e.g., further software installations must be possible over the network or from some USB key; so it must include some tools to support the network, etc.).
- A system with looked-after architecture, quality and ease of use. I know you said "consistent" but I expressly want to state what I consider the most important aspects of that notion.

From these statements we can draw a lot of implications, assuming people agree with this list. Below, I'll limit myself to those in connection with the need to easily build from source and externalizing components out of base (for all software, not LLVM specifically).

Having a port for the 'src'-required compiler(s) or other components of base certainly has advantages (the machinery to fetch code, apply patches and build already exists), but in this light also has drawbacks.

First, it becomes less straightforward to work (inspect, modify, etc.) with the code to be built. If following the traditional port model, the pristine code would be fetched from upstream, and our modifications would be made available through ports tree patches. Producing the final source then requires applying the patches (the equivalent of "make patch" in ports) for each "remote" component one wants to inspect. If, instead, the project hosts already tuned source code in the form of tar balls, then there is the question of how to fetch them and unpack all files in a tree that again should preferably make sense to developers.

Second, both approaches easily lose the ability to inspect the history of each remote component if not done properly. At worst, it is always possible to ship a '.git' directory, or equivalent, but recording somewhere the commit identifier and having the infrastructure to either clone the repo at the proper commit, or switch an existing checkout to it, would be much better. However, these approaches miss one important point: We don't necessarily need all upstream's histories, although it would help, but instead probably require being able to quickly check out which code was shipped with which release, or is part of some -STABLE or -CURRENT, now or at some earlier point in time, and upstream repositories obviously can't answer these questions on their own.

So I think that, if we actually massively switch to external components, be they LLVM or not, we are going to need a fair amount of tooling to keep a smooth developer/security auditer/you-name-it experience with FreeBSD (those that require the source code and navigating through it). It may be that this is not deemed necessary for externalizing LLVM alone (I'm not a LLVM integrator, nor developer), but I doubt we can smoothly generalize the use of external components without it.

Thanks and regards.

[David Chisnall]

I would add that Poudriere has *great* support for this. It can create VM images and a bunch of related things. It can do this from a src tree *or* a binary release, and can then build and install packages from ports and add other overlays on top.

I wish the release engineering team would adopt it, it’s far easier for other people to reproduce the flow with Poudriere than with the release scripts.

It’s not perfect, but it’s better than any alternative I’ve seen (for any OS).

David

[Daniel Eischen]

Post by David Chisnall
I would add that Poudriere has *great* support for this. It can create VM images and a bunch of related things. It can do this from a src tree *or* a binary release, and can then build and install packages from ports and add other overlays on top.
I wish the release engineering team would adopt it, it’s far easier for other people to reproduce the flow with Poudriere than with the release scripts.
It’s not perfect, but it’s better than any alternative I’ve seen (for any OS).

+1

The ports tree never seems to have the defaults I want, it's great for setting and remembering those options, then building based on those.

--
DE

[Tomek CEDRO]

Olivier touched nicely core of FreeBSD design - logic, simplicity,
coherence, self sufficiency, repeatability, compartmentalization, and
many more - all those come from current / historical design of the src
NOT the proposed one that leads to avalanche of new problems and
complications mentioned by Olivier. I fully agree here.

SRC contains both source code of the operating system and all tools to
build it in one place. No matter what version / medium /
transportation / storage (release / commit / package / zip / tar / git
/ svn / whatever). This keeps things simple coherent and self
sufficient. Zero dependencies nightmare for the user.

Yes, building from sources is the foundation of Open-Source, and SRC
that keeps both operating system along with versioned patched ready to
use build tools in one place is so non-problematic that people seem to
have a problem with that.

What is more that simple change in design (aka "lets remove build
tools from src") sounds not even like fake innovation but more like
purposeful diversion - divide core component and then divide divided
until project stops and people leave because others do it better
anyways.

I humbly assume this discussion to be more of a thought provoking
experiment rather than a serious technical proposition :-)

[Poul-Henning Kamp]

Post by Olivier Certner

We need to find a contemporary and useful answer to "What is FreeBSD?"
Delivering a single consistent userland with the kernel has stood
us well for three decades, and we should stick with that.

- A system that is easy to build and tweak in practice (for developers at the very least).

But what are the boundaries of this "system" of which you talk ?

I am more or less responsible for nearly two hundred computers
running FreeBSD right now. Only two of those have zero ports/packages
installed, one monitors my floor heating system, the other firewalls
some old crap.

To me "src+kernel" is just the foundations.

"A system" is what people build on top of the foundations.
--
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20
***@FreeBSD.ORG | TCP/IP since RFC 956
FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Olivier Certner]

Post by Poul-Henning Kamp

Post by Olivier Certner
- A system that is easy to build and tweak in practice (for developers at the very least).

But what are the boundaries of this "system" of which you talk ?

In the quote to which you responded, "system" simply designates FreeBSD, or the "base system" to be even more explicit.

I have not accurately described what this "system" should include, simply because I don't believe it can be delimited precisely ex abrupto in all generality, as this depends on a lot of factors (amount of people; what they are willing and able to work on; which upstreams project exist, can be used, and are cooperative; technology evolutions; our collective project direction; etc.).

That's why I gave some high-level view of what I think the project is and should remain, and without which its nature would be considerably altered (I'd really like to hear other's points of view). And I contend that this view is more important than the precise detail of whether we have this or that as "base".

However, according to that view, I'm more or less coming to the conclusion that FreeBSD should include the compiler. Which doesn't mean its code has to *live* in the 'src' git repository, nor does it mean we cannot rebuild FreeBSD without using pre-packaged binary or an earlier build from source.

Post by Poul-Henning Kamp
- A system that is self-sufficient, in the sense that, once installed on a machine, one can set it up to do whatever one would like without having to boot something else (e.g., further software installations must be possible over the network or from some USB key; so it must include some tools to support the network, etc.).
- A system that is easy to build and tweak in practice (for developers at the very least).

I would add that "self-sufficient" also means that it can be easily built entirely from source from itself and other commonly available systems. And also, as developed at the end of my previous mail, that it is easy to obtain a workable view of all the source.

Post by Poul-Henning Kamp
I am more or less responsible for nearly two hundred computers
running FreeBSD right now. Only two of those have zero ports/packages
installed, one monitors my floor heating system, the other firewalls
some old crap.
To me "src+kernel" is just the foundations.
"A system" is what people build on top of the foundations.

That's one possible view of a system, which is consumer-oriented rather than developer or software-project-oriented. And I could return the question to you: What's the limit of your "system"? One computer? Your network? A group of machines owned by someone? Or accomplishing some special task? Does it include the network (routers, cables, etc.)? Does it include your ISP? These different groupings can always be seen as foundations of other, bigger ones. Without more context, I don't see how this foundation/system distinction is going to be fruitful.

By "system", or FreeBSD base, I mean code written, overseen and/or managed by the people comprising the FreeBSD project, with the goals listed in my previous mail. I think that's the fundamental difference with ports: Ports are... ports of software developed externally (which doesn't prevent some porters to contribute to upstream), not necessarily tied to FreeBSD's code, not having the same quality insurance, nor the same coding policies or release schedules, and not collectively maintained by members of the project (at least, we do not promise that; of course, members are individually free to participate to whatever upstream project they like).

There is no denying that for lots of (most?) tasks, being able to run these ports is a pre-requisite, it's just that we are not responsible for their development. What porters are responsible for is ensuring that we can build and run these softwares on top of FreeBSD, with varying degrees of quality (depending on the size of upstream, the number of porters for each project, their level of engagement, etc.). This sometimes entails participating to upstream development, but does not require setting, e.g., the full architecture or roadmap of these projects.

So, while we as a group are not living in an isolated island for sure, and, e.g., should strive to maintain good working relationship with other projects, we are not (and cannot) be the developers or maintainers of all useful software. But we have to be so for software that we consider part of FreeBSD.

Of course, code that is our own (provided, again, that it is needed to fulfill the high-level principles), i.e., for which there's no upstream and no changes happen that aren't done or fully reviewed by project members (well, this may be slightly idealistic, but hey...), belongs to 'src'.

Finally, we have the kind of middle-ground comprising upstream projects, such as LLVM, which AFAICT are growing in number. As long as we consider them to be needed in FreeBSD ("base") according to the high-level principles, we are responsible for them functioning correctly and consistently with the rest of the system, even if we are not the primary developers. So, we also "own" this code, albeit differently to code entirely in our hands. And it should be maintained collectively. From commits and internal discussions, I can identify more or less correctly the (small) group of FreeBSD people working on LLVM in FreeBSD. If, for some reason, these people would stop working on it, we would have a problem, to which I expect the project as a whole would respond with actively trying to find a solution (some would invest more time to work on it, and/or we would try to find external people who can, or we would switch to another toolchain, etc.). If, for some reason, let's say, Firefox would lose enough of its developers to have its viability threatened, I don't expect us (as a project) to react to that as Firefox is not part of base. This doesn't exclude having a project response to the difficulty of *porting* some software that we deem very important for the ecosystem, occasionally or in a longer term fashion.

This is only my view, but I have the feeling it is implicitly shared by many. I'm very interested in hearing whether it is more or less conform to reality and what people think about it as of now and for FreeBSD's future.

That said, I also think this debate is mostly independent from the possibility of building FreeBSD without having to rebuild a full compiler every time. It is however relevant to the proposal of removing LLVM's code from 'src', as this has bad consequences (according to the principles I listed) that we probably can, and should, remedy with tooling, although, as the saying goes, the daemon is in the details.

Thanks and regards.

[Alexander Leidinger]

Am 2024-09-10 11:05, schrieb Olivier Certner:

[your point of view]

Post by Olivier Certner
This is only my view, but I have the feeling it is implicitly shared by
many. I'm very interested in hearing whether it is more or less
conform to reality and what people think about it as of now and for
FreeBSD's future.

I share a similar point of view. The difference is ... below.

Post by Olivier Certner
That said, I also think this debate is mostly independent from the
possibility of building FreeBSD without having to rebuild a full
compiler every time. It is however relevant to the proposal of
removing LLVM's code from 'src', as this has bad consequences
(according to the principles I listed) that we probably can, and
should, remedy with tooling, although, as the saying goes, the daemon
is in the details.

IMO it doesn't hurt to move the toolchain out of src, as long as we have
a port, e.g. freebsd/toolchain-for-version-X (or whatever we want to
call it), which is buildable on all supported release we want to support
the build of FreeBSD-X. As we support to build FreeBSD also on Linux and
OS-X, we would also need to have support to make it not too hard to
build said toolchain on those systems.

And I go even further, I would not only mind to install a port for a
compiler, I would also be ok to install a port to install a tool which
handles the build (replacing make in src). I would be ok to run "pkg
install/update X Y Z" to be able to do "cd /usr/src; do_build freebsd &&
do_install freebsd". I fully agree that it should be easy to build the
base system, to do what we do today, and I think is would be beneficial
if apart from "make installworld" if we could also update via "pkg
upgrade". I think we could provide what we provide currently while at
the same time reap some benefits from having those parts managed in
ports. IMO there are a lot of benefits with such an approach, be it
system hardening, compile time, and more. I would not want to move
everything to ports (toolchain yes, build system yes, now that we
switched to DMA: sendmail yes, openssl/kerberos no -> too much
integration with other parts), but I would not mind having a ports-like
approach for src separate from ports (I don't know if pkgbase fits this
or not, I haven't looked at it, but if I can define a pkg repo for base
and populate it from e.g. "make pkgrepo DESTDIR=xxx" or similar and then
just need to run pkg upgrade that would be nice).

Bye,
Alexander.

[Olivier Certner]

IMO it doesn't hurt to move the toolchain out of src (...)
(...) I would not want to move everything to ports (...), but I would not
mind having a ports-like approach for src (...)

To clarify and be sure we are on the same page, I also don't mind what you describe *provided* there is appropriate tooling to easily:
- Get all the code part of base, and not only for building it.
- Navigate its history, both for code changes but also integration in base for upstream projects (upstream history is nice, but not enough).

In other words, I insist on having the same ease of use that we have today with everything in a single repository. Being able to just build base *is not enough*. Else, moving things to ports is going to cause important pain for several use cases such as code inspection and auditing, collaborative maintenance of code moved to ports, understanding why/whether some changes have impact on some components, system consistency, etc.

(I don't know if pkgbase fits this or not, I haven't looked at it, but if I
can define a pkg repo for base and populate it from e.g. "make pkgrepo
DESTDIR=xxx" or similar and then just need to run pkg upgrade that would be
nice).

I haven't used it personally, but from what I read, I expect what you describe to be exactly its main supported use case.

Thanks and regards.

[Warner Losh]

Post by Alexander Leidinger

Post by Olivier Certner

IMO it doesn't hurt to move the toolchain out of src (...)
(...) I would not want to move everything to ports (...), but I would not
mind having a ports-like approach for src (...)

To clarify and be sure we are on the same page, I also don't mind what
- Get all the code part of base, and not only for building it.
- Navigate its history, both for code changes but also integration in
base for upstream projects (upstream history is nice, but not enough).
In other words, I insist on having the same ease of use that we have
today with everything in a single repository. Being able to just build
base *is not enough*. Else, moving things to ports is going to cause
important pain for several use cases such as code inspection and
auditing, collaborative maintenance of code moved to ports,
understanding why/whether some changes have impact on some components,
system consistency, etc.

I consider src, ports, and docs as separate repos, not as one repo (we
can branch them separately, that's the point of distinction for me).
With that POV I do not think it is pragmatic to have the toolchain in
the same repo. I understand that it's convenient and less painful, and
pragmatic in regards to what you have mentioned, to keep it within a
repo in our control (like now, just in parallel to ports/src/doc).

And if we had it separate, we could also play the submodule game. I'm
hesitant to do that with 3rd party repos, but for one we control it could
give us the best of both worlds... too bad submodules have so many rough
edges... but a small number, well controlled with appropriate automation
could have a high useful to painful ratio.

Warner

Post by Alexander Leidinger
Bye,
Alexander.
--

[Alexander Leidinger]

Post by Olivier Certner

IMO it doesn't hurt to move the toolchain out of src (...)
(...) I would not want to move everything to ports (...), but I would not
mind having a ports-like approach for src (...)

To clarify and be sure we are on the same page, I also don't mind what
- Get all the code part of base, and not only for building it.
- Navigate its history, both for code changes but also integration in
base for upstream projects (upstream history is nice, but not enough).
In other words, I insist on having the same ease of use that we have
today with everything in a single repository. Being able to just build
base *is not enough*. Else, moving things to ports is going to cause
important pain for several use cases such as code inspection and
auditing, collaborative maintenance of code moved to ports,
understanding why/whether some changes have impact on some components,
system consistency, etc.

I consider src, ports, and docs as separate repos, not as one repo (we
can branch them separately, that's the point of distinction for me).
With that POV I do not think it is pragmatic to have the toolchain in
the same repo. I understand that it's convenient and less painful, and
pragmatic in regards to what you have mentioned, to keep it within a
repo in our control (like now, just in parallel to ports/src/doc).

Bye,
Alexander.