adding new flua libraries

Discussion:

(too old to reply)

Mark Johnston

2024-09-05 22:51:53 UTC

FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's
intended for use by the base system, and so far has a few consumers, but
not many so far. (As an aside, flua's intended scope is not totally
clear to me. Is it only for use by the base system? What compatibility
guarantees does it provide, if any?)

A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
available, but they don't provide enough to make flua useful as a
general purpose programming tool. It lacks interfaces for interacting
with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
standard programming facilities (e.g., classes, higher-order functions,
etc.). Here I'm mostly interested in discussing the former.

I think flua could be a very useful alternative to shell scripts and C
code where performance is not critical. It's very good at wrapping C
interfaces and thus could be used to make FreeBSD features (jails,
bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
don't want to interact with C.

It's a lot of work to build up a set of flua modules that provide enough
functionality to be generally useful. My feeling is that the easiest
way to get there is to wrap C interfaces directly (to the extent that
that's possible, but it's usually easy) and expose them in flua as a
stable interface. Libraries written in pure Lua (or other languages
that interoperate well with Lua) could be built on top of that.

I'm inclined to start by wrapping libc and libsys interfaces in a manner
similar to luaposix. There, the namespace is partitioned by C headers,
so posix.unistd contains identifiers from unistd.h, posix.sys.stat
contains identifiers from sys/stat.h, and so on. In fact, flua already
has a small subset of luaposix's functionality. Wrapping C interfaces
isn't much fun, but it's easy, and it saves us having to come up with
names and interfaces (naming things is hard), and helps ensure that the
glue code is relatively small and doesn't impose a large testing burden.
Moreover, C interfaces don't change much and are subject to
well-understood compatibility constraints, which should mean that Lua
interfaces built on top of them are subject to the same constraints.

Assuming there's general agreement on that approach, the question I'd
really like to answer is, what should the namespace look like? It would
be useful to provide a posix module compatible with luaposix, but that
obviously won't contain FreeBSD-specific functionality.

I propose having a top-level "freebsd" namespace for all modules
implemented in the base system, excluding posix and contrib libraries
which already define a Lua interface (libucl is the one example I see so
far; we could import sqlite bindings as well). Then, if we follow
luaposix's convention, we could have freebsd.sys.capsicum.* for
Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
wrappers, and so on. The posix module could simply provide a subset of
freebsd.*.

Maybe it's better to separate C wrappers from native Lua modules though,
so it could be better to have freebsd.c.sys.capsicum.*, etc., and
provide higher-level interfaces for FreeBSD features under freebsd.pf,
freebsd.zfs, freebsd.jail, etc.. I'm not really sure.

In the interest of prompting discussion a bit, I posted some patches to
add some example wrappers to flua:
https://reviews.freebsd.org/D46553
https://reviews.freebsd.org/D46554
https://reviews.freebsd.org/D46556

Does anyone have opinions on anything I've brought up above? I'm pretty
happy to write a lot of this glue code or find ways to automate it, as
I've already done a fair bit of it, but it's hard to make progress
without having some rigourous conventions for the flua namespace (again,
naming things is hard).

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Kyle Evans

2024-09-05 23:11:47 UTC

Permalink

Only by the base system (hence the rename and hiding it in
/usr/libexec), no guarantees except for modules we've re-implemented
subsets of- if a conflicting name exists in ports, its use must be
compatible with that.

Post by Mark Johnston
A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
available, but they don't provide enough to make flua useful as a
general purpose programming tool. It lacks interfaces for interacting
with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
standard programming facilities (e.g., classes, higher-order functions,
etc.). Here I'm mostly interested in discussing the former.
I think flua could be a very useful alternative to shell scripts and C
code where performance is not critical. It's very good at wrapping C
interfaces and thus could be used to make FreeBSD features (jails,
bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
don't want to interact with C.
It's a lot of work to build up a set of flua modules that provide enough
functionality to be generally useful. My feeling is that the easiest
way to get there is to wrap C interfaces directly (to the extent that
that's possible, but it's usually easy) and expose them in flua as a
stable interface. Libraries written in pure Lua (or other languages
that interoperate well with Lua) could be built on top of that.
I'm inclined to start by wrapping libc and libsys interfaces in a manner
similar to luaposix. There, the namespace is partitioned by C headers,
so posix.unistd contains identifiers from unistd.h, posix.sys.stat
contains identifiers from sys/stat.h, and so on. In fact, flua already
has a small subset of luaposix's functionality. Wrapping C interfaces
isn't much fun, but it's easy, and it saves us having to come up with
names and interfaces (naming things is hard), and helps ensure that the
glue code is relatively small and doesn't impose a large testing burden.
Moreover, C interfaces don't change much and are subject to
well-understood compatibility constraints, which should mean that Lua
interfaces built on top of them are subject to the same constraints.
Assuming there's general agreement on that approach, the question I'd
really like to answer is, what should the namespace look like? It would
be useful to provide a posix module compatible with luaposix, but that
obviously won't contain FreeBSD-specific functionality.
I propose having a top-level "freebsd" namespace for all modules
implemented in the base system, excluding posix and contrib libraries
which already define a Lua interface (libucl is the one example I see so
far; we could import sqlite bindings as well). Then, if we follow
luaposix's convention, we could have freebsd.sys.capsicum.* for
Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
wrappers, and so on. The posix module could simply provide a subset of
freebsd.*.

I think that's fine. Ideally, we probably just import luaposix
wholesale now once we get the bootstrap flua module situation sorted and
we can just re-export those implementations into the freebsd.* namespace
similar to how we discussed luaposix was doing things today, unless
those interfaces are weird (I don't remember them being problematic,
though).

Post by Mark Johnston
Maybe it's better to separate C wrappers from native Lua modules though,
so it could be better to have freebsd.c.sys.capsicum.*, etc., and
provide higher-level interfaces for FreeBSD features under freebsd.pf,
freebsd.zfs, freebsd.jail, etc.. I'm not really sure.
In the interest of prompting discussion a bit, I posted some patches to
https://reviews.freebsd.org/D46553
https://reviews.freebsd.org/D46554
https://reviews.freebsd.org/D46556

Will take a look when I get a bit of time.

Post by Mark Johnston
Does anyone have opinions on anything I've brought up above? I'm pretty
happy to write a lot of this glue code or find ways to automate it, as
I've already done a fair bit of it, but it's hard to make progress
without having some rigourous conventions for the flua namespace (again,
naming things is hard).

We should also figure out where to document this stuff (policy, mostly-
we have a 3lua section for libraries themselves).

Thanks,

Kyle Evans

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Kyle Evans

2024-09-06 02:39:39 UTC

Permalink

Post by Mark Johnston

Only by the base system (hence the rename and hiding it in /usr/libexec), no
guarantees except for modules we've re-implemented subsets of- if a
conflicting name exists in ports, its use must be compatible with that.

I think that's fine. Ideally, we probably just import luaposix wholesale
now once we get the bootstrap flua module situation sorted and we can just
re-export those implementations into the freebsd.* namespace similar to how
we discussed luaposix was doing things today, unless those interfaces are
weird (I don't remember them being problematic, though).

Could you please elaborate on the bootstrap module problem?

We build a bootstrap version of flua that was initially used to
accommodate, e.g., `make sysent` after the conversion of makesyscalls.
That'll become more important for the Linux/macOS builds if (when) some
or all sysent artifacts are generated as part of the build as well.

The problem is that we don't setup the bootstrap flua to be able to do
loadable modules (and we don't bootstrap any modules, for that matter),
so we need to fix that before we further adopt it.

Post by Mark Johnston
Regarding importing luaposix, one potential problem there is with
interopability of userdata types, mainly the treatment of file
descriptors (which I would want to use in, e.g., freebsd.sys.capsicum).
For example, in the reviews linked below I went with a userdata type for
file descriptors so that they can be garbage-collected automatically,
but luaposix doesn't do that AFAICS. To me that's a bit unsavoury but
I'm not sure how important it really is. I do think that we could quite
quickly and easily reimplement most of luaposix, for what it's worth.

Ahh, that makes sense. I don't feel that strongly about importing the
rest of luaposix wholesale, so I'm perfectly happy to drop that idea; as
you note, it's easy enough to reimplement it.

Post by Mark Johnston

Will take a look when I get a bit of time.

Thank you kindly.

We should also figure out where to document this stuff (policy, mostly- we
have a 3lua section for libraries themselves).

Yep, I added a hacky posix.3lua in one of the reviews linked above. A
man page for flua itself is probably the next step.

Excellent!

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Mark Johnston

2024-09-06 00:34:29 UTC

Permalink

Only by the base system (hence the rename and hiding it in /usr/libexec), no
guarantees except for modules we've re-implemented subsets of- if a
conflicting name exists in ports, its use must be compatible with that.

I think that's fine. Ideally, we probably just import luaposix wholesale
now once we get the bootstrap flua module situation sorted and we can just
re-export those implementations into the freebsd.* namespace similar to how
we discussed luaposix was doing things today, unless those interfaces are
weird (I don't remember them being problematic, though).

Could you please elaborate on the bootstrap module problem?

Regarding importing luaposix, one potential problem there is with
interopability of userdata types, mainly the treatment of file
descriptors (which I would want to use in, e.g., freebsd.sys.capsicum).
For example, in the reviews linked below I went with a userdata type for
file descriptors so that they can be garbage-collected automatically,
but luaposix doesn't do that AFAICS. To me that's a bit unsavoury but
I'm not sure how important it really is. I do think that we could quite
quickly and easily reimplement most of luaposix, for what it's worth.

Will take a look when I get a bit of time.

Thank you kindly.

We should also figure out where to document this stuff (policy, mostly- we
have a 3lua section for libraries themselves).

Yep, I added a hacky posix.3lua in one of the reviews linked above. A
man page for flua itself is probably the next step.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Baptiste Daroussin

2024-09-06 09:29:20 UTC

Permalink

Post by Mark Johnston
FreeBSD ships a Lua 5.4 implementation, flua, in the base system. It's
intended for use by the base system, and so far has a few consumers, but
not many so far. (As an aside, flua's intended scope is not totally
clear to me. Is it only for use by the base system? What compatibility
guarantees does it provide, if any?)
A few flua modules wrapping FreeBSD libraries (e.g., libjail) are
available, but they don't provide enough to make flua useful as a
general purpose programming tool. It lacks interfaces for interacting
with the system (e.g., libc/libsys/libutil/etc wrappers) as well as
standard programming facilities (e.g., classes, higher-order functions,
etc.). Here I'm mostly interested in discussing the former.
I think flua could be a very useful alternative to shell scripts and C
code where performance is not critical. It's very good at wrapping C
interfaces and thus could be used to make FreeBSD features (jails,
bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
don't want to interact with C.
It's a lot of work to build up a set of flua modules that provide enough
functionality to be generally useful. My feeling is that the easiest
way to get there is to wrap C interfaces directly (to the extent that
that's possible, but it's usually easy) and expose them in flua as a
stable interface. Libraries written in pure Lua (or other languages
that interoperate well with Lua) could be built on top of that.
I'm inclined to start by wrapping libc and libsys interfaces in a manner
similar to luaposix. There, the namespace is partitioned by C headers,
so posix.unistd contains identifiers from unistd.h, posix.sys.stat
contains identifiers from sys/stat.h, and so on. In fact, flua already
has a small subset of luaposix's functionality. Wrapping C interfaces
isn't much fun, but it's easy, and it saves us having to come up with
names and interfaces (naming things is hard), and helps ensure that the
glue code is relatively small and doesn't impose a large testing burden.
Moreover, C interfaces don't change much and are subject to
well-understood compatibility constraints, which should mean that Lua
interfaces built on top of them are subject to the same constraints.
Assuming there's general agreement on that approach, the question I'd
really like to answer is, what should the namespace look like? It would
be useful to provide a posix module compatible with luaposix, but that
obviously won't contain FreeBSD-specific functionality.
I propose having a top-level "freebsd" namespace for all modules
implemented in the base system, excluding posix and contrib libraries
which already define a Lua interface (libucl is the one example I see so
far; we could import sqlite bindings as well). Then, if we follow
luaposix's convention, we could have freebsd.sys.capsicum.* for
Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
wrappers, and so on. The posix module could simply provide a subset of
freebsd.*.
Maybe it's better to separate C wrappers from native Lua modules though,
so it could be better to have freebsd.c.sys.capsicum.*, etc., and
provide higher-level interfaces for FreeBSD features under freebsd.pf,
freebsd.zfs, freebsd.jail, etc.. I'm not really sure.
In the interest of prompting discussion a bit, I posted some patches to
https://reviews.freebsd.org/D46553
https://reviews.freebsd.org/D46554
https://reviews.freebsd.org/D46556
Does anyone have opinions on anything I've brought up above? I'm pretty
happy to write a lot of this glue code or find ways to automate it, as
I've already done a fair bit of it, but it's hard to make progress
without having some rigourous conventions for the flua namespace (again,
naming things is hard).

I like all of what I read and I am fully aligned.

What I don't see here, is I think we should have dynamic modules libucl, fbsd &
friends are not dynamic and the reviews I see for the freebsd module reviews,
this is still bundled.

my proposal for unbundling, I need kldload for a project of mine, so I did:
https://reviews.freebsd.org/D46558

Best regards,
Bapt

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Kyle Evans

2024-09-06 14:45:19 UTC

Permalink

Post by Baptiste Daroussin

Right, this is an artifact of how flua's evolved that we need to move
away from. We didn't have module support at the beginning- that came
later, but we didn't really move things out into modules. We still
can't move some things because of the bootstrap flua problem I mentioned
elsewhere (doing so would block work to do `make sysent` type work as
part of the build as it would break cross-builds), but at least
libucl/fbsd would be fine.

Post by Baptiste Daroussin
https://reviews.freebsd.org/D46558
Best regards,
Bapt

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

Warner Losh

2024-09-06 14:48:59 UTC

Permalink

Post by Baptiste Daroussin

I like all of what I read and I am fully aligned.
What I don't see here, is I think we should have dynamic modules libucl,

fbsd &

Post by Baptiste Daroussin
friends are not dynamic and the reviews I see for the freebsd module

reviews,

Post by Baptiste Daroussin
this is still bundled.

Yea, if the sysent.lua code doesn't use modules, would we still have the
same
bootstrapping issues? Today, I don't think here's anything but vanilla lua
code
in there.

Warner

Post by Baptiste Daroussin

Post by Baptiste Daroussin
my proposal for unbundling, I need kldload for a project of mine, so I
https://reviews.freebsd.org/D46558
Best regards,
Bapt

Warner Losh

2024-09-06 14:47:21 UTC

Permalink

To date, it's only provided for FreeBSD provided things. That means there's
close coordination between what we provide and it. In general, it should
remain
compatible, but I think we've taken some liberties when it wasn't widely
used
to, for example, go from 5.3 to 5.4 w/o providing a fully compatible 5.3
env.

Yea, LUA is a language you can build OO concepts on, but doesn't
directly provide more than the building blocks. Most librairies I've seen
either pick a set of wrappers, or expand the wrappers in-line. We should
pick one: I'm partial to the provide a common set of wrappers model
when the time comes.

Post by Mark Johnston
I think flua could be a very useful alternative to shell scripts and C
code where performance is not critical. It's very good at wrapping C
interfaces and thus could be used to make FreeBSD features (jails,
bhyve, ctl, pf, zfs, dtrace, ...) more accessible to developers who
don't want to interact with C.

Yes. Last summer's GSoC code to test all the boot loader operations
went great in lua until the very end when the desire to test all 50
combinations in parallel rather than serially (to reduce the run time
from sneaking up on an hour to about a minute) couldn't do it in Lua,
so things were rewritten in Python...

Post by Mark Johnston
It's a lot of work to build up a set of flua modules that provide enough
functionality to be generally useful. My feeling is that the easiest
way to get there is to wrap C interfaces directly (to the extent that
that's possible, but it's usually easy) and expose them in flua as a
stable interface. Libraries written in pure Lua (or other languages
that interoperate well with Lua) could be built on top of that.

Yes. I tend to agree, though I think we may want to find a way to
provide some things written in Lua and some in C for these
wrapping.

Post by Mark Johnston
I'm inclined to start by wrapping libc and libsys interfaces in a manner
similar to luaposix. There, the namespace is partitioned by C headers,
so posix.unistd contains identifiers from unistd.h, posix.sys.stat
contains identifiers from sys/stat.h, and so on. In fact, flua already
has a small subset of luaposix's functionality. Wrapping C interfaces
isn't much fun, but it's easy, and it saves us having to come up with
names and interfaces (naming things is hard), and helps ensure that the
glue code is relatively small and doesn't impose a large testing burden.
Moreover, C interfaces don't change much and are subject to
well-understood compatibility constraints, which should mean that Lua
interfaces built on top of them are subject to the same constraints.

Generally I agree, though with the caveat that I think posix module
tends to have too-deep a namespace, and I'm hesitant about using
that for FreeBSD, given how much clutter it would add to the lua code.

Post by Mark Johnston
Assuming there's general agreement on that approach, the question I'd
really like to answer is, what should the namespace look like? It would
be useful to provide a posix module compatible with luaposix, but that
obviously won't contain FreeBSD-specific functionality.
I propose having a top-level "freebsd" namespace for all modules
implemented in the base system, excluding posix and contrib libraries
which already define a Lua interface (libucl is the one example I see so
far; we could import sqlite bindings as well). Then, if we follow
luaposix's convention, we could have freebsd.sys.capsicum.* for
Capsicum-related syscalls and constants, freebsd.sys.event.* for kevent
wrappers, and so on. The posix module could simply provide a subset of
freebsd.*.

I think that may be too hierarchical and provide little benefit. What would
we have in freebsd.* that would conflict with freebsd.sys.*? I barely see
the
benefit for the posix module, but I think it was a mistake to do it
there... but
we're stuck with the design.

And depending on how compatible we want to be with the posix module,
this may tie our hands a bit in providing more info, or info in a better
format if we try to make it strictly speaking, a subset with the implication
that the behavior is identical.

I think that's going even further into the weeds. There's no reason
that freebsd.zfs couldn't provide the zfs system calls. freebsd.jail the
jail
ones, etc, in addition to the higher levels. We're not likely going to nest
multiple modules that do jail things on top of a base jail system call-only
module. I'm having trouble seeing the benefit for all the extra clutter
this would bring to the lua code we write.

Post by Mark Johnston
In the interest of prompting discussion a bit, I posted some patches to
https://reviews.freebsd.org/D46553
https://reviews.freebsd.org/D46554
https://reviews.freebsd.org/D46556
Does anyone have opinions on anything I've brought up above? I'm pretty
happy to write a lot of this glue code or find ways to automate it, as
I've already done a fair bit of it, but it's hard to make progress
without having some rigourous conventions for the flua namespace (again,
naming things is hard).

We should agree on the namespace stuff, then put the conversation in our
rearview mirror. :).

But generally, I love the idea and any comments about how to design "the
crinkly bits on the fjords"
shouldn't be taken to my not liking the fjords.

Warner

with apologies to douglas adams