TrueNAS Build: Meteora

Introduction

Parts List

Type	Brand	Model	Description
Motherboard	Supermicro	X12STL-F	Micro-ATX
Processor	Intel	Xeon E-2324G	4-Cores @ 3.1GHz (4.60GHz)
Memory	Supermicro	MEM-DR432MD-EU32	4x32GB DDR4 3200 ECC UDIMM
Storage (Boot)	Supermicro	SSD-DM032-SMCMVN1	32GB SATA DOM
Storage (SLOG)	Intel	Optane P1600X	4x118GB, 1760/1050 MB/s, 410000/243000 IOPS, 1292 TBW, 2x ZFS Mirrors
Storage1	Samsung	PM893	4x3.84TB, 550/520MB/s, 98000/30,000 IOPS, ZFS Mirror
Storage2
Network Card	Chelsio	T520-CR	PCIe 3.0 x8, Dual SFP+ (in PCIe Slot 7)
Host Bus Adapter	LSI	9300-8i	PCIe 3.0 x8, IT Mode (in PCIe Slot 5)
Storage Expander	Intel	RES3FV288	PCIe x4 (Power Only) (in PCIe Slot 4)
Storage Adapter	Supermicro	AOC-SHG3-4M2P	Quad NVMe M.2 SSD PCIe 8x Adapter (in PCIe Slot 6)

Diagrams

Drive Placement/Layout

SSD (Pool1 vDev1/Mirror)	SSD (Pool1 vDev1/Mirror)	SSD (Pool1 vDev2/Mirror)	SSD (Pool1 - vDev2/Mirror)
HDD (Pool2 vDev1)	HDD (Pool2 vDev1)	HDD (Pool2 vDev2)	HDD (Pool2 vDev2)
HDD (Pool2 vDev1)	HDD (Pool2 vDev1)	HDD (Pool2 vDev2)	HDD (Pool2 vDev2)
HDD (Pool2 vDev1)	HDD (Pool2 vDev1)	HDD (Pool2 vDev2)	HDD (Pool2 vDev2)
HDD (Pool2 vDev1)	HDD (Pool2 vDev1)	HDD (Pool2 vDev2)	HDD (Pool2 vDev2)

Pool1 consists of 4x 3.84TB SSDs, which are broken up into 2 vdevs, each with two mirrored drives.
Pool2 consists of 16x HDDs, which are broken up into 2 vdevs, each with two parity disks (RAIDZ2).

Motherboard (Supermicro X12STL-F)

Motherboard Layout Diagram

Motherboard System Block Diagram

HBA and Expander Connection Diagram

Build Notes

PCIe Card Placement Issues

The initial build utilized the following PCIe slots and arrangement for the cards:

PCIe Slot	Card
PCH Slot4 - PCIe 3.0 x2 in	Storage Expander - Intel RES3FV288
CPU Slot5 - CPU PCIe 4.0 x8	Host Bus Adapter - LSI 9300-8i
CPU Slot6 - CPU 4.0 x8 in x16	Storage Adapter - Supermicro AOC-SHG3-4M2P + 4x Intel Optane P1600X M.2 drives
CPU Slot7 - CPU 4.0 x4 in x8	Network Card - Chelsio T520-CR

Using the PCIe card order referenced above, I encountered difficulties in obtaining an IP address for the network card (Chelsio T520-CR). While all the other cards functioned flawlessly, the red status indicator LED on the network card continuously blinked red during my testing. This indicated that the card itself was not loading a driver, despite having the TrueNAS tunables configured, tunables that were used in a previously working build.

Upon conducting a more detailed examination of the board, I observed a disparity between PCIe Slot7 and PCIe Slots 4-6 in terms of the presence of capacitors. Specifically, I noticed that PCIe Slot7 lacked the capacitors found in the other slots. While I wasn't certain if this was directly related to the issue at hand, it prompted me to reevaluate the card placement in the build.

Slot 4-7 (left to right):

By swapping the cards between PCIe Slot6 and PCIe Slot7, I was greeted with a fully functional Chelsio T520-CR card in PCIe Slot 6. Furthermore, the red status indicator LED ceased blinking, indicating that the driver successfully loaded and I was able to finally obtain an IP address. The final build now incorporates the following PCIe card arrangement:

PCIe Slot	Card
PCH Slot4 - PCH PCIe 3.0 x2 in	Storage Expander - Intel RES3FV288
CPU Slot5 - CPU PCIe 4.0 x8	Host Bus Adapter - LSI 9300-8i
CPU Slot6 - CPU PCIe 4.0 x8 in x16	Network Card - Chelsio T520-CR
CPU Slot7 - CPU PCIe 4.0 x4 in x8	Storage Adapter - Supermicro AOC-SHG3-4M2P + 4x Intel Optane P1600X M.2 drives

This adjustment proved to be satisfactory without compromising any functionality. The storage adapter (Supermicro AOC-SHG3-4M2P) is now successfully installed in PCIe Slot 7, which remains a PCIe 4.0 x4 slot capable of delivering impressive data speeds of up to 8GB/s. These speeds surpass the capabilities of the 4x Intel Optane P1600X M.2 drives that are currently housed in the storage adapter. It's worth noting that the 4x Intel Optane P1600X M.2 drives will be divided into two mirrored vdevs, with each vdev dedicated as a SLOG device for the two pools running in this build.

Additional Airflow

Unknown Network Interface

After completing a new installation of TrueNAS, I discovered the presence of an unknown network interface labeled ue0 - the interface "Type" was "Unknown". Subsequently, I found that this interface is associated with the Supermicro IPMI host interface, which can easily be disabled in the IPMI configuration settings.