How to explain high memory consumption of a jail after all large processed in it have finished?

[Yuri]

I noticed that when the port lang/rust is building in the poudriere jail
the memory consumption of the host system remains high all the way into
the packaging phase when the pkg-static process is the only active
process and it consumes a very little memory.


During build a lot of memory is consumed, which is understandable. The
system remains at ~500MB of free memory through the build process,
according to top(1).


But once the build is finished, poudriere goes into the "packaging"
phase which only runs a small pkg-static process that compresses the
built files. pkg-static is the only active process in the poudriere jail.


What looks strange to me is that the host system's memory consumption
remains high through the "packaging" phase which itself is low in
memory, and only goes down when the jail is destroyed.


How to explain the high memory consumption of a jail after all large
presses have finished?



Thanks,

Yuri




--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Pat Maddox]

Post by Yuri
I noticed that when the port lang/rust is building in the poudriere jail
the memory consumption of the host system remains high all the way into
the packaging phase when the pkg-static process is the only active
process and it consumes a very little memory.

How much of the memory is inactive? -Pat


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Millard]

Yuri <yuri_at_FreeBSD.org> wrote on

Post by Yuri
I noticed that when the port lang/rust is building in the poudriere jail
the memory consumption of the host system remains high all the way into
the packaging phase when the pkg-static process is the only active
process and it consumes a very little memory.
During build a lot of memory is consumed, which is understandable. The
system remains at ~500MB of free memory through the build process,
according to top(1).
But once the build is finished, poudriere goes into the "packaging"
phase which only runs a small pkg-static process that compresses the
built files. pkg-static is the only active process in the poudriere jail.
What looks strange to me is that the host system's memory consumption
remains high through the "packaging" phase which itself is low in
memory, and only goes down when the jail is destroyed.
How to explain the high memory consumption of a jail after all large
presses have finished?

You do not give much information about configuration
properties that contribute to memory usage patterns
and how to interpret them.

How many FreeBSD cpus does the system have?

How much RAM?
What do the likes of the boot output (dmesg -a) show for:
real memory = ??? (??? MB)
avail memory = ??? (??? MB)

What does the likes of:

# swapinfo -m

show for "1M-blocks" (Total) once SWAP is fully set up?


ZFS in use? (Any tuning/configuration of note?)
Only UFS in use (no ZFS anyway)?


/usr/local/etc/poudriere.conf (or analogous command line
settings):

How many poudriere bulk builders are there that have been
or are active prior to or during things? (The likes of
PARALLEL_JOBS= . . . can contribute to the answer.)

What is USE_TMPFS= . . . set to?
(all? yes? wrkdir? data? localbase? no? some list of such?)

What is TMPFS_BLACKLIST= . . . set to (if anything)?
(What is TMPFS_BLACKLIST_TMPDIR= . . . set to [if anything]?)

What is ALLOW_MAKE_JOBS= . . . set to?
What is ALLOW_MAKE_JOBS_PACKAGES= . . . set to (if anything)?


/usr/local/etc/poudriere.d/*make.conf (or analogous command
line settings):

What is MAKE_JOBS_NUMBER_LIMIT= . . .set to (if anything)?
(Or analogous settings: there are several related ones.)

What is TMPFS_LIMIT= . . . set to (if anything)?
What is MAX_MEMORY= . . . set to (if anything)?

What is CCACHE_DIR= . . . set up (if anything)?

Outside such configuration files:

There are other things that do not stand by themselves well.
For example: "system remains at ~500MB of free memory". Is
the amount of SWAP space usage varying? For "Mem:"
Active varying?
Inact varying?
Laundry varying? (Is top even showing it? If not: zero.)
Wired varying?
(You are indicating Free is roughly constant at ~500MB for
a type of context.)

(Example figures for comparisons/contrasts could prove
interesting.)

ZFS's ARC Total varying? (if zfs ARC is in use; contributes
mostly to Wired)


Other things for your context:

When rust is building, is it its builder the only active
builder?

What does the likes of:

# df -mt tmpfs | sort -k6,6

show during, say, the packaging stage?


Note:

tmpfs RAM usage can stick around longer than one might
expect, including while a builder is idle between jobs
or after its last job.

One thing for rust builds is that they have a large
file system usage with materials that stick around
even during the package phase. Thus, there are
consequences for RAM+SWAP space competition/usage if
such end up being handled via tmpfs for rust builds.


There are contexts for which (Active+Inact+Laundry+Wired+Free)
shrinks over time compared to "avail memory" --even to the
point of failure/hangup. I've no clue what is at issue if
significant such shrinkage is happening for your context. (No
claim such is happening but it is something that data might
show.)

Note: top's "Buf" overlaps with data in the the other
categories and would cause a double-counting of some RAM
usage if included.


===
Mark Millard
marklmi at yahoo.com



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Alexander Leidinger]

Post by Yuri
I noticed that when the port lang/rust is building in the poudriere
jail the memory consumption of the host system remains high all the way
into the packaging phase when the pkg-static process is the only active
process and it consumes a very little memory.
During build a lot of memory is consumed, which is understandable. The
system remains at ~500MB of free memory through the build process,
according to top(1).
But once the build is finished, poudriere goes into the "packaging"
phase which only runs a small pkg-static process that compresses the
built files. pkg-static is the only active process in the poudriere jail.
What looks strange to me is that the host system's memory consumption
remains high through the "packaging" phase which itself is low in
memory, and only goes down when the jail is destroyed.
How to explain the high memory consumption of a jail after all large
presses have finished?

That's not enough information to answer it, but my crystal ball hints at
an interaction between nullfs/zfs. Most probably it's not the
jail-teardown itself which frees the mem, but the unmounting of some
nullfs mounts.

Depending on the version of FreeBSD you are using, you could try if
"sysctl vfs.nullfs.cache_vnodes=0" changes this behavior (I'm not saying
that this is a "solution" to the "issue" you see, I simply say this may
change the behavior).

Bye,
Alexander.

[Dimitry Andric]

I noticed that when the port lang/rust is building in the poudriere jail the memory consumption of the host system remains high all the way into the packaging phase when the pkg-static process is the only active process and it consumes a very little memory.
During build a lot of memory is consumed, which is understandable. The system remains at ~500MB of free memory through the build process, according to top(1).
But once the build is finished, poudriere goes into the "packaging" phase which only runs a small pkg-static process that compresses the built files. pkg-static is the only active process in the poudriere jail.
What looks strange to me is that the host system's memory consumption remains high through the "packaging" phase which itself is low in memory, and only goes down when the jail is destroyed.
How to explain the high memory consumption of a jail after all large presses have finished?

Most likely this is tmpfs. Look at the USE_TMPFS= setting poudriere.conf. I usually set it to "localbase", otherwise my rather low memory boxes all blow up. Or you can just use "no" to turn it off entirely, which might make it a little slower.

-Dimitry



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Chris]

I noticed that when the port lang/rust is building in the poudriere jail the
memory consumption of the host system remains high all the way into the
packaging
phase when the pkg-static process is the only active process and it consumes
a
very little memory.
During build a lot of memory is consumed, which is understandable. The
system
remains at ~500MB of free memory through the build process, according to
top(1).
But once the build is finished, poudriere goes into the "packaging" phase
which
only runs a small pkg-static process that compresses the built files.
pkg-static
is the only active process in the poudriere jail.
What looks strange to me is that the host system's memory consumption
remains high
through the "packaging" phase which itself is low in memory, and only goes
down
when the jail is destroyed.
How to explain the high memory consumption of a jail after all large presses
have finished?

Apologies in advance if already addressed, but I'm way behind on my email.
I'm gonna guess the compression stage is responsible. My experience shows it
really chews through CPU cycles, and as a result memory.

--Chris

Thanks,
Yuri