Question of installing over SAS controller and USB flash

[a***@icloud.com]

Hi all,
a newbie in FreeBSD here trying to install a RAIDZ2 on a server with mps SAS controller with drives which are not visible in the boot order in BIOS.
So, I cannot install directly on the drives, despite that the installer "sees" them, because the BIOS does not see the controller and its drives as option.
On the other hand, I have a card reader with a flash card on it (4gigs) from which I can boot and where I can have my /boot partition with the kernel.
I've a lot of questions and I still cannot find the answers to most of them, despite going through the handbook, the boot process explanation, the installation process handbook and a lot of more reading.


* How should the flash card be partitioned and with what mountpoints? The server is BIOS capable.
* How should the rest of the drives be partitioned with ZFS and the mountpoints? The standard configuration from the installer on RAIDZ2 is good for me.

I've experience with Linux but is not helping a lot here. The setup looks quite different .... probably because of the ZFS.

Any help, links, ideas and comments will be much appreciated!


Cyberly yours,
deragoth

[Warner Losh]

Post by a***@icloud.com
Hi all,
a newbie in FreeBSD here trying to install a RAIDZ2 on a server with mps
SAS controller with drives which are not visible in the boot order in BIOS.
So, I cannot install directly on the drives, despite that the installer
"sees" them, because the BIOS does not see the controller and its drives as
option.

Have you enabled the BIOS on the mps card itself? Otherwise it won't
install a BIOS fragment to run at boot, and the drives won't be seen.

On the other hand, I have a card reader with a flash card on it (4gigs)

Post by a***@icloud.com
from which I can boot and where I can have my /boot partition with the
kernel.
I've a lot of questions and I still cannot find the answers to most of
them, despite going through the handbook, the boot process explanation, the
installation process handbook and a lot of more reading.
- How should the flash card be partitioned and with what mountpoints?
The server is BIOS capable.
- How should the rest of the drives be partitioned with ZFS and the
mountpoints? The standard configuration from the installer on RAIDZ2 is
good for me.
I've experience with Linux but is not helping a lot here. The setup looks
quite different .... probably because of the ZFS.
Any help, links, ideas and comments will be much appreciated!

That's a workaround, but you should be able to enable the BIOS part of the
mps/mpr cards and boot off them. I've done that before.

Warner

Post by a***@icloud.com
Cyberly yours,
deragoth

[Miroslav Lachman]

Hey Warner,
the box is very featureLESS (an old Xyratex HS-1235T box, as far as I
researched it).
T2029_QR v1.3.ai (hackspace.org.uk)
<https://wiki.london.hackspace.org.uk/w/images/b/b2/HS-1235T-ATX_Quick_Reference_Sheet.pdf>
(5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.com)
<https://www.reddit.com/r/homelab/comments/8loge6/xyratex_094403703_motherboard_mystery_solved/>
Through the POST process and the BIOS there is no option to get into the
SAS controller and enable its BIOS. I would love to install FreeBSD
directly on the drives and not to deal with the flash card, but I
suppose the lack of controller BIOS is the reason the box comes with
reader and card directly.

[..]

This depends on at what stage the computer starts to see the disk behind
the SAS controller. I had an old HP server that only saw one disk out of
four in the BIOS, so it couldn't boot from RAIDZ, but it was enough to
make a 10GB partition on each disk as a 4 way ZFS mirror, boot from that
and use the rest of each disk for the main RAIDZ storage (4x 8TB).

Maybe it would have been enough to use a USB flash drive for the
freebsd-boot partition in GPT scheme and then continue boot from the
disks on the SAS controller.

gpart create -s GPT da0
gpart bootcode -b /boot/pmbr da0
gpart add -b 40 -s 512 -t freebsd-boot da0
gpart bootcode -p /boot/zfsgptboot -i 1 da0

As long as the computer boots from the installer CD, FreeBSD with RAIDZ
can be installed on the disks behind the SAS controller without any
modification, only the USB flash drive must provide the boot.

If the USB flash drive must also contain a UFS /boot partition, then
copy an image that can be downloaded from the web as a "mini-memstick"
to the USB flash drive
https://www.freebsd.org/releases/13.2R/announce/ (use dd according to
the instructions)
Or again, create a complete installation from the FreeBSD installer on
the USB flash drive.

But the installation to big drives with RAIDZ should be done normally
from the FreeBSD installer. No modifications needed.

Kind regards
Miroslav Lachman



--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-***@muc.de

[Warner Losh]

There's not a FreeBSD port/pkg for lsiutil, but I'ved used it at $WORK for
things.

https://github.com/thomaslovell/LSIUtil

has an archive of sources. I've not used this at all, and it hasn't been
ported to FreeBSD in this repo.
But I know it allows one to turn on/off the POST / BIOS in the card, which
is what you need. It runs
under Linux for sure (again, don't know what packages are available there).
I don't think I can share
what I have for $WORK with the public since I think we got it in a way that
doesn't let me disclose the
sources.

One might also be able move the code lsiutil that enables it to mpsutil.
It's going to be sending a raw
command to the device. It shouldn't be too hard....

Warner

Hey Warner,
the box is very featureLESS (an old Xyratex HS-1235T box, as far as I
researched it).
T2029_QR v1.3.ai (hackspace.org.uk)
<https://wiki.london.hackspace.org.uk/w/images/b/b2/HS-1235T-ATX_Quick_Reference_Sheet.pdf>
(5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.com)
<https://www.reddit.com/r/homelab/comments/8loge6/xyratex_094403703_motherboard_mystery_solved/>
Through the POST process and the BIOS there is no option to get into the
SAS controller and enable its BIOS. I would love to install FreeBSD
directly on the drives and not to deal with the flash card, but I suppose
the lack of controller BIOS is the reason the box comes with reader and
card directly.
As I'm new to FreeBSD and ZFS I'm struggling to set up the partitioning. I
1. Partition the entire FLASH card (4gigs) on MBR with freebsd-ufs and
mount /boot. As I've read there should not be ESP (efi) partition as this
is flash, thus MBR, which is "legacy" and without ESP.
2. Partition one of the drives (3tera) with GPT, freebsd-zfs stripe
and mount /
3. When the system is up and running, add the rest of the drives (11
more) to the pool and change the pool to RAIDZ2. However I'm not sure this
is possible, to change the type of the pool without erasing everything.
Logical thinking says it is not possible.
4. After all of these comes the mounting of filesystems and creating
fstab to continue the installation, which is full darkness at the moment,
as I'm still struggling with the initial start and not have read anything
on that.
All of these should be done in shell with the gpart utility during install
and I cannot figure out, how to make points 2 and 3 on one go having
directly the RAIDZ2.
------------------------------
*Sent:* Wednesday, October 11, 2023 00:20
*Subject:* Re: Question of installing over SAS controller and USB flash
Hi all,
a newbie in FreeBSD here trying to install a RAIDZ2 on a server with mps
SAS controller with drives which are not visible in the boot order in BIOS.
So, I cannot install directly on the drives, despite that the installer
"sees" them, because the BIOS does not see the controller and its drives as
option.
Have you enabled the BIOS on the mps card itself? Otherwise it won't
install a BIOS fragment to run at boot, and the drives won't be seen.
On the other hand, I have a card reader with a flash card on it (4gigs)
from which I can boot and where I can have my /boot partition with the
kernel.
I've a lot of questions and I still cannot find the answers to most of
them, despite going through the handbook, the boot process explanation, the
installation process handbook and a lot of more reading.
- How should the flash card be partitioned and with what mountpoints?
The server is BIOS capable.
- How should the rest of the drives be partitioned with ZFS and the
mountpoints? The standard configuration from the installer on RAIDZ2 is
good for me.
I've experience with Linux but is not helping a lot here. The setup looks
quite different .... probably because of the ZFS.
Any help, links, ideas and comments will be much appreciated!
That's a workaround, but you should be able to enable the BIOS part of the
mps/mpr cards and boot off them. I've done that before.
Warner
Cyberly yours,
deragoth

[a***@icloud.com]

Hey Miroslav,
thank you for the help.
I've tried with the first part of your suggestion - gpart bootcode, but that was not enough. I suppose I do need the /boot on the flash card, as I've installed Linux and put the /boot "folder" on the flash and everything was working flawlessly.
I'll try your suggestion with the memstick. Just a question there.
Do I need to make the "gpart bootcode" part if I'm creating the memstick, or the memstick has everything needed to boot from it, i.e. it has the bootcodes?

Thanks again for your help and willingness to waste your time on a noob questions!

Cyberly yours,
Nikolay Balov


________________________________
From: owner-freebsd-***@freebsd.org <owner-freebsd-***@freebsd.org> on behalf of Miroslav Lachman <***@quip.cz>
Sent: Wednesday, October 11, 2023 16:49
To: ***@icloud.com <***@icloud.com>; Warner Losh <***@bsdimp.com>
Cc: freebsd-***@FreeBSD.org <freebsd-***@FreeBSD.org>
Subject: Re: Question of installing over SAS controller and USB flash

Hey Warner,
the box is very featureLESS (an old Xyratex HS-1235T box, as far as I
researched it).
T2029_QR v1.3.ai (hackspace.org.uk)
<https://wiki.london.hackspace.org.uk/w/images/b/b2/HS-1235T-ATX_Quick_Reference_Sheet.pdf>
(5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.com)
<https://www.reddit.com/r/homelab/comments/8loge6/xyratex_094403703_motherboard_mystery_solved/>
Through the POST process and the BIOS there is no option to get into the
SAS controller and enable its BIOS. I would love to install FreeBSD
directly on the drives and not to deal with the flash card, but I
suppose the lack of controller BIOS is the reason the box comes with
reader and card directly.

[..]

This depends on at what stage the computer starts to see the disk behind
the SAS controller. I had an old HP server that only saw one disk out of
four in the BIOS, so it couldn't boot from RAIDZ, but it was enough to
make a 10GB partition on each disk as a 4 way ZFS mirror, boot from that
and use the rest of each disk for the main RAIDZ storage (4x 8TB).

Maybe it would have been enough to use a USB flash drive for the
freebsd-boot partition in GPT scheme and then continue boot from the
disks on the SAS controller.

gpart create -s GPT da0
gpart bootcode -b /boot/pmbr da0
gpart add -b 40 -s 512 -t freebsd-boot da0
gpart bootcode -p /boot/zfsgptboot -i 1 da0

As long as the computer boots from the installer CD, FreeBSD with RAIDZ
can be installed on the disks behind the SAS controller without any
modification, only the USB flash drive must provide the boot.

If the USB flash drive must also contain a UFS /boot partition, then
copy an image that can be downloaded from the web as a "mini-memstick"
to the USB flash drive
https://www.freebsd.org/releases/13.2R/announce/ (use dd according to
the instructions)
Or again, create a complete installation from the FreeBSD installer on
the USB flash drive.

But the installation to big drives with RAIDZ should be done normally
from the FreeBSD installer. No modifications needed.

Kind regards
Miroslav Lachman