Migrating to LLVM binutils tools (ar, nm, addr2line, etc.)

[Ed Maste]

FreeBSD migrated from GNU binutils to versions from ELF Tool Chain,
starting in 2014. At that time there were no usable LLVM versions of
those tools, but they have been developing rapidly since then. Now I
think it may be prudent to migrate to the LLVM tools where they exist,
for both functionality and maintainability reasons.

I'd like to allow use of link-time optimization (LTO) in the FreeBSD
base system. LTO runs optimization passes over the entire executable
(or library) at link time and thus allows for more effective
optimization than when performed on individual compilation units.

When using LTO object files (.o) including those contained in static
library archives (.a) contain LLVM IR bitcode rather than target
object code. This means that utilities that operate on object files
need to support LLVM IR; we currently use a number of bespoke tools
and ones obtained from ELF Tool Chain that do not have this support.

Alex Richardson also pointed out that asan (address sanitizer)
produces a useful backtrace only if addr2line is llvm-symbolizer.

Like ELF Tool Chain the LLVM tools aim for command line and output
format compatibility with GNU binutils, although there are a few minor
differences. Where these cause a material issue (breaking a port or
eliminating required functionality) we can submit LLVM bugs and work
on patches.

In the past we provided build knobs to control individual utilities
(e.g. WITH_LLD_IS_LD); I'd like to avoid perpetuating that here. It
seems individual knobs (WITH_LLVM_AR_IS_AR, WITH_LLVM_NM_IS_NM,
WITH_LLVM_SYMBOLIZER_IS_ADDR2LINE etc.) will introduce extra
complexity without adding much value.

Alex is working on a patch now and will follow up shortly, but I
wanted to email the list as a heads-up, and see if there are any
comments or concerns.

Potential next steps are:
- Introduce new build knob
- Iterate on exp-runs and call for testing
- Switch to LLVM tools by default
- Major release (14.0)
- Retire knob, leaving only the LLVM implementation.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Richardson]

Post by Ed Maste
FreeBSD migrated from GNU binutils to versions from ELF Tool Chain,
starting in 2014. At that time there were no usable LLVM versions of
those tools, but they have been developing rapidly since then. Now I
think it may be prudent to migrate to the LLVM tools where they exist,
for both functionality and maintainability reasons.
I'd like to allow use of link-time optimization (LTO) in the FreeBSD
base system. LTO runs optimization passes over the entire executable
(or library) at link time and thus allows for more effective
optimization than when performed on individual compilation units.
When using LTO object files (.o) including those contained in static
library archives (.a) contain LLVM IR bitcode rather than target
object code. This means that utilities that operate on object files
need to support LLVM IR; we currently use a number of bespoke tools
and ones obtained from ELF Tool Chain that do not have this support.
Alex Richardson also pointed out that asan (address sanitizer)
produces a useful backtrace only if addr2line is llvm-symbolizer.
Like ELF Tool Chain the LLVM tools aim for command line and output
format compatibility with GNU binutils, although there are a few minor
differences. Where these cause a material issue (breaking a port or
eliminating required functionality) we can submit LLVM bugs and work
on patches.
In the past we provided build knobs to control individual utilities
(e.g. WITH_LLD_IS_LD); I'd like to avoid perpetuating that here. It
seems individual knobs (WITH_LLVM_AR_IS_AR, WITH_LLVM_NM_IS_NM,
WITH_LLVM_SYMBOLIZER_IS_ADDR2LINE etc.) will introduce extra
complexity without adding much value.
Alex is working on a patch now and will follow up shortly, but I
wanted to email the list as a heads-up, and see if there are any
comments or concerns.

Thanks for writing this up, I've posted the initial patch as
https://reviews.freebsd.org/D31060.

Post by Ed Maste
- Introduce new build knob
- Iterate on exp-runs and call for testing
- Switch to LLVM tools by default
- Major release (14.0)
- Retire knob, leaving only the LLVM implementation.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Shawn Webb]

Post by Ed Maste
FreeBSD migrated from GNU binutils to versions from ELF Tool Chain,
starting in 2014. At that time there were no usable LLVM versions of
those tools, but they have been developing rapidly since then. Now I
think it may be prudent to migrate to the LLVM tools where they exist,
for both functionality and maintainability reasons.
I'd like to allow use of link-time optimization (LTO) in the FreeBSD
base system. LTO runs optimization passes over the entire executable
(or library) at link time and thus allows for more effective
optimization than when performed on individual compilation units.
When using LTO object files (.o) including those contained in static
library archives (.a) contain LLVM IR bitcode rather than target
object code. This means that utilities that operate on object files
need to support LLVM IR; we currently use a number of bespoke tools
and ones obtained from ELF Tool Chain that do not have this support.
Alex Richardson also pointed out that asan (address sanitizer)
produces a useful backtrace only if addr2line is llvm-symbolizer.
Like ELF Tool Chain the LLVM tools aim for command line and output
format compatibility with GNU binutils, although there are a few minor
differences. Where these cause a material issue (breaking a port or
eliminating required functionality) we can submit LLVM bugs and work
on patches.
In the past we provided build knobs to control individual utilities
(e.g. WITH_LLD_IS_LD); I'd like to avoid perpetuating that here. It
seems individual knobs (WITH_LLVM_AR_IS_AR, WITH_LLVM_NM_IS_NM,
WITH_LLVM_SYMBOLIZER_IS_ADDR2LINE etc.) will introduce extra
complexity without adding much value.
Alex is working on a patch now and will follow up shortly, but I
wanted to email the list as a heads-up, and see if there are any
comments or concerns.
- Introduce new build knob
- Iterate on exp-runs and call for testing
- Switch to LLVM tools by default
- Major release (14.0)
- Retire knob, leaving only the LLVM implementation.

Hey Ed,

As background for anyone curious, HardenedBSD switched to using
llvm-ar, llvm-nm, and llvm-objdump by default years ago as part of the
work to start integrating Cross-DSO CFI.

We've noticed one small, but important, issue with llvm-ar (which is
also the same underlying program as llvm-ranlib) in some behavior that
doesn't match ELF Toolchain's ar/ranlib (which I'll call elftc-ar).

For most cases, when elftc-ar fails, it does not set the exitcode to
non-zero. This tricks the ports tree to continue to build a port where
elftc-ar actually errored.

llvm-ar does the right thing in exiting with a non-zero exit code on
error.

However, due to this discrepency in behavior, certain ports that cause
an error condition when calling ar/ranlib continue to build when
elftc-ar is used, but fail to build when llvm-ar is used.

I'm thinking that I'll report this same issue to the ELF Toolchain
folks since elftc-ar really should exit with a non-zero exitcode on
failure.

I've just now hacked llvm-ar to behave the same as elftc-ar[0] and
will do a poudriere bulk run soon.

I'll report back my status with the ELF Toolchain notification and the
poudriere run as soon as I have more info.

[0]:
https://git.hardenedbsd.org/hardenedbsd/HardenedBSD/-/commit/5bdcc54a23f05883f55e895da49726955fa8b07b

Thanks,

[Ed Maste]

Post by Shawn Webb
For most cases, when elftc-ar fails, it does not set the exitcode to
non-zero. This tricks the ports tree to continue to build a port where
elftc-ar actually errored.

Thanks for the notice - I didn't see this email until now, but saw the
related discussion on HardenedBSD's IRC channel this morning.

Note that FreeBSD ar is different from ELF Tool Chain's ar; the latter
is a fork of FreeBSDs, and there are some distinct bug fixes or
improvements in each version that are not present in the other. Here
FreeBSD's ar is buggy, and llvm-ar, GNU ar, and ELF Tool Chain's ar
all return a non-zero exit code.

Code review with fix for FreeBSD's ar: https://reviews.freebsd.org/D31402
Exp-run request: https://bugs.freebsd.org/257599


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Shawn Webb]

Post by Ed Maste

Post by Shawn Webb
For most cases, when elftc-ar fails, it does not set the exitcode to
non-zero. This tricks the ports tree to continue to build a port where
elftc-ar actually errored.

Thanks for the notice - I didn't see this email until now, but saw the
related discussion on HardenedBSD's IRC channel this morning.
Note that FreeBSD ar is different from ELF Tool Chain's ar; the latter
is a fork of FreeBSDs, and there are some distinct bug fixes or
improvements in each version that are not present in the other. Here
FreeBSD's ar is buggy, and llvm-ar, GNU ar, and ELF Tool Chain's ar
all return a non-zero exit code.
Code review with fix for FreeBSD's ar: https://reviews.freebsd.org/D31402
Exp-run request: https://bugs.freebsd.org/257599

I was mistaken about which ar was errant. I apologize. Thank you
very much for the clarification.

Thanks again,

[Ed Maste]

Post by Ed Maste
- Iterate on exp-runs and call for testing

The exp-run is open in 258872. Perl initially failed to build, which
caused thousands of ports to be skipped. That issue was fixed by markj
in 27f35b7dd418.

TCL and PostgreSQL are the next two ports that cause a significant
number of skipped dependencies. Both failures are due to the use of
"strip -x" against an archive (or static library). Strip -x is used to
strip all non-global symbols, but in both cases there are non-global
symbols that cannot be stripped, and so LLVM strip reports an error. I
am not sure why these two (and a few others) want to install a
stripped archive in the first place. TCL now has a patch in the ports
tree (thanks gahr@) while waiting on a more portable upstream fix, and
PostgreSQL has a fix committed upstream. The same issue affects
textproc/sxml, but isn't a dependency for much (i.e., does not cause
any significant number of skipped ports).

The next issue affects science/q and shells/bash-static, and is caused
by mixing GNU objcopy and LLVM strip. There's a good description of
the presumed issue in
https://github.com/llvm/llvm-project/issues/53948#issuecomment-1518486410.
A bug was introduced in GNU objcopy about a year and a half ago, and
fixed last month. Due to this bug GNU objcopy produced invalid ELF
objects, with relocation sections that linked to the wrong symbol
table. GNU objcopy and strip are permissive and silently ignore the
invalid input, white LLVM objcopy and strip reject it with an error.
The combination of GNU and LLVM tools is probably undesired; I suspect
what's happening here is the port USES binutils, that doesn't override
strip and so the system strip is still used (PR270663).

The remaining issue is in sysutils/stressdisk, PR270962. This appears
to be a problem between LLVM strip and code built by cgo, with an
upstream bug at https://github.com/llvm/llvm-project/issues/53999.

A couple of other leaf ports may be broken when WITH_LLVM_BINUTILS is set.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Ed Maste]

Post by Ed Maste
FreeBSD migrated from GNU binutils to versions from ELF Tool Chain,
starting in 2014. At that time there were no usable LLVM versions of
those tools, but they have been developing rapidly since then. Now I
think it may be prudent to migrate to the LLVM tools where they exist,
for both functionality and maintainability reasons.

For the rest of the context see the thread at
https://lists.freebsd.org/archives/freebsd-hackers/2021-July/000111.html

Post by Ed Maste
- Introduce new build knob

This was done by Alex Richardson in commit 021385aba562.

Post by Ed Maste
- Iterate on exp-runs and call for testing

The exp-run is PR258872. There are five open bugs for ports failing to
build with LLVM_BINUTILS:

1. PR270421 graphics/librsvg2-rust

This needs to be fixed in the port -- it combines ports LLVM with base
system nm / ar, and they're not compatible for LTO.

2. PR270962 sysutils/stressdisk

This is an issue in llvm-strip -
https://github.com/llvm/llvm-project/issues/53999. For FreeBSD ports
the short-term workaround is probably to have it use GNU binutils
instead

3. PR270963 textproc/sxml

The PR has a patch that addresses the issue, but has some issues that
need to be addressed before being committed.

4. PR278172 lang/racket*
5. PR278895 lang/ghc

It's not clear to me what's happening for these two. Likely switch to
GNU binutils for them.

Post by Ed Maste
- Switch to LLVM tools by default
- Major release (14.0)

This clearly did not happen for FreeBSD 14.0, but I plan to do this before 15.0.

Post by Ed Maste
- Retire knob, leaving only the LLVM implementation.

The knob would remain in place for at least one release; this will not
happen before 16.0.


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de