TarFS

[Mark Saad]

Hello All,
I was wondering if anyone has started to play with tarfs in FreeBSD 14 ?
It appears to puke with larger tarballs. For example , if I fetch a copy of
ports or pkgsrc and decompress it to just a posix tar archive I can't get
it to mount.

***@mono:/home/nonesuch # file ports.tar
ports.tar: POSIX tar archive
***@mono:/home/nonesuch # mount -t tarfs ./ports.tar /tarfs/ports
mount: ./ports.tar: Inappropriate file type or format

***@mono:/home/nonesuch # file pkgsrc.tar
pkgsrc.tar: POSIX tar archive
***@mono:/home/nonesuch # mount -t tarfs pkgsrc.tar /tarfs/ports
mount: pkgsrc.tar: Invalid argument

***@mono:/home/nonesuch # ls -alsh ports.tar
585312 -rw-r--r-- 1 nonesuch nonesuch 571M Mar 22 01:01 ports.tar
***@mono:/home/nonesuch # ls -alsh pkgsrc.tar
567360 -rw-r--r-- 1 root nonesuch 554M Mar 16 01:29 pkgsrc.tar

Before I open a bug here, is this expected ?

[Dag-Erling Smørgrav]

I was wondering if anyone has started to play with tarfs in FreeBSD 14?

I wrote it, does that count?

It appears to puke with larger tarballs.

On the contrary, one of the use cases I tested was buildworld from a
tarball of the source tree, which it handled just fine.

For example, if I fetch a copy of ports or pkgsrc and decompress it to
just a posix tar archive I can't get it to mount.
ports.tar: POSIX tar archive
mount: ./ports.tar: Inappropriate file type or format

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst
ports.tar.zst 47 MB 5431 kBps 09s
% sudo mount -rt tarfs $PWD/ports.tar.zst /mnt
mount: /home/des/ports.tar.zst: Inappropriate file type or format
% dmesg | grep tarfs
tarfs_alloc_one: unsupported global extended header at 0
% zcat ports.tar.zst | hexdump -C | head -3
00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65 |pax_global_heade|
00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |r..............|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |...............|

This is a non-standard extension header emitted by `git archive` which
contains metadata about the git tree from which the archive was created.
I believe you can safely strip it off (zcat ports.tar.zst | dd bs=1k
skip=1 >ports.tar), but you'll run into issues with very long paths in
devel/electron*. I will take a closer look when I find the time.

Note that tarfs can mount tarballs compressed with zstd, but performance
will be poor unless you create a multi-frame archive (try using --zstd
--options=zstd:frame-per-file,zstd:min-frame-size=65536).

DES

[Eugene Grosbein]

Post by Mark Saad
Again good work. C an we expect to see a libarchive-fs next :)

We have sysutils/archivemount in ports.



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Mark Saad]

Post by Dag-Erling Smørgrav

I was wondering if anyone has started to play with tarfs in FreeBSD 14?

I wrote it, does that count?

It appears to puke with larger tarballs.

On the contrary, one of the use cases I tested was buildworld from a
tarball of the source tree, which it handled just fine.

Dag let me just say , great work. I really like this.

Post by Dag-Erling Smørgrav

For example, if I fetch a copy of ports or pkgsrc and decompress it to
just a posix tar archive I can't get it to mount.
ports.tar: POSIX tar archive
mount: ./ports.tar: Inappropriate file type or format

% fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/ports/ports.tar.zst
ports.tar.zst 47 MB 5431 kBps
09s
% sudo mount -rt tarfs $PWD/ports.tar.zst /mnt
mount: /home/des/ports.tar.zst: Inappropriate file type or format
% dmesg | grep tarfs
tarfs_alloc_one: unsupported global extended header at 0
% zcat ports.tar.zst | hexdump -C | head -3
00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65
|pax_global_heade|
00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|r...............|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
This is a non-standard extension header emitted by `git archive` which
contains metadata about the git tree from which the archive was created.
I believe you can safely strip it off (zcat ports.tar.zst | dd bs=1k
skip=1 >ports.tar), but you'll run into issues with very long paths in
devel/electron*. I will take a closer look when I find the time.
Note that tarfs can mount tarballs compressed with zstd, but performance
will be poor unless you create a multi-frame archive (try using --zstd
--options=zstd:frame-per-file,zstd:min-frame-size=65536).
DES

So I am not sure exactly what tarfs is yelling about with pkgsrc.tar , but
I wonder if it's a by-product of netbsd using pax .
NetBSD tar's dont work , OpenBSD stuff does and they use a ustar format.

***@mono:/home/nonesuch # cat pkgsrc.tar |hexdump -C |head -3
00000000 70 6b 67 73 72 63 2f 00 00 00 00 00 00 00 00 00
|pkgsrc/.........|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*
***@mono:/home/nonesuch # cat ports.tar |hexdump -C |head -3
00000000 70 61 78 5f 67 6c 6f 62 61 6c 5f 68 65 61 64 65
|pax_global_heade|
00000010 72 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|r...............|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|


***@mono:/home/nonesuch # cat xenocara.tar |hexdump -C |head -3
00000000 2e 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
*


Two other questions ( I have not tried to read the code , please forgive my
assumptions ) behind the scenes are the tarballs extracted over some fs,
or mounted directly and read ?
Second are there any contradictions with tarfs ; can they be nfs exported,
mounted on a tmpfs etc .

Again good work. C an we expect to see a libarchive-fs next :)

Post by Dag-Erling Smørgrav
--

[Dag-Erling Smørgrav]

Post by Mark Saad
Two other questions ( I have not tried to read the code , please
forgive my assumptions ) behind the scenes are the tarballs extracted
over some fs, or mounted directly and read ?

It scans the archive, builds an index of its contents, and reads what
you ask for when you ask for it. No additional disk space is consumed.
This is why compressed archives can be very slow.

Post by Mark Saad
Second are there any contradictions with tarfs ; can they be nfs
exported, mounted on a tmpfs etc .

They can be exported, but I'm not sure what you mean by “mounted on a
tmpfs”.

DES

[Warner Losh]

Post by Dag-Erling Smørgrav

Post by Mark Saad
Two other questions ( I have not tried to read the code , please
forgive my assumptions ) behind the scenes are the tarballs extracted
over some fs, or mounted directly and read ?

It scans the archive, builds an index of its contents, and reads what
you ask for when you ask for it. No additional disk space is consumed.
This is why compressed archives can be very slow.

Post by Mark Saad
Second are there any contradictions with tarfs ; can they be nfs
exported, mounted on a tmpfs etc .

They can be exported, but I'm not sure what you mean by “mounted on a
tmpfs”.

I think he means does it work witg unionfs.

Side question: how hard is it to add and offset of tarball mount option?

Warner

Post by Dag-Erling Smørgrav
DES
--

[Dag-Erling Smørgrav]

Post by Warner Losh

Post by Dag-Erling Smørgrav
They can be exported, but I'm not sure what you mean by “mounted on a
tmpfs”.

I think he means does it work witg unionfs.

There's no reason why it shouldn't, although I haven't tested it.
Unlike exportability, which requires explicit support in each and every
filesystem, it's handled transparently by VFS.

Post by Warner Losh
Side question: how hard is it to add and offset of tarball mount option?

It shouldn't be hard, since all I/O goes through tarfs_io_read(), but
why would you want to?

DES

[Warner Losh]

--------

Post by Dag-Erling Smørgrav

Post by Warner Losh
Side question: how hard is it to add and offset of tarball mount option?

It shouldn't be hard, since all I/O goes through tarfs_io_read(), but
why would you want to?

To skip headers TarFS does not grok ?

It would make it possible to have all the files for an application in an
ELF section formatted as a tarball... AppImage
https://docs.appimage.org/reference/specification.html is one thing that
does in Linux... with user mount enabled the wrapper could mount the app's
files in a TarFS... Though Linux has a generic "at a given offset" flag to
mount which might be better....

And a big thanks for TarFS from me too!
Definitely cool.

Warner

Poul-Henning Kamp | UNIX since Zilog Zeus 3.20

FreeBSD committer | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.