Promxox PVE w/ TrueNAS Core VM

I've been running my homelab for a few months now with a Dell R720 enterprise server running TrueNAS Scale as the primary NAS and as a virtualization server. My backup NAS is a Dell Optiplex 5090 MT also running TrueNAS Scale. The R720 runs Plex Media Server and Heimdall as applications in TrueNAS Scale.

The Optiplex runs a Windows VM under TrueNAS for BlueIris. This machine includes a single 10TB HDD serving as video storage and as a replication target for the primary NAS.

I would really like to use the TrueNAS application library, however support is limited and many of the applications that I've tried simply do not run or crash. I fully acknowledge that this result may be my own limited ability. But several TrueNAS blog posts talk about how the TrueNAS application catalog is not well maintained. I will move to a different solution for now.

The subject of this gist is my next Home Lab project: build a replacement virtualization server that also doubles as a NAS. The existing TruenNAS server shall become the backup and will not support any virtulization. Ultamately, the R720 will be replaced with something that is lower power to serve as the backup NAS.

The future of the BlueIris host machine is still an open question. I'm leaning to installing Proxmox on this machine too. I've read that a Proxmox ZFS pool can be the target of a TrueNAS remote replication. More on this topic later.

Requirements for this build

1. Repurpose enterprise server hardware with 8 HDD bays and ECC memory
2. Proxmox as the virtualization server
3. TrueNAS core running in a Proxmox VM

Historically I've run TrueNAS core as my NAS it has been reliable for me. Also, it is the recomended version for production deployments. So for now my experiment with TrueNAS Scale is over. I will use TrueNAS and Proxmox for what each does best. In this build, TrueNAS shall operate only as a NAS. None of its applications shall be deployed and there will not be any VMs running with it.

Applications that will run as Proxmox LXC containers:

1. Plex Media Server
2. Docker

Running Docker in an LXC container is not recomended by the Proxmox support team. However. I've been reading many blog posts from users that do this successfully. The advantage of the LXC container is mamory allocation; the VM is assigned a fixed allocation and the LXC container is dynamically assigned as needed. This dynamic allocation seems like a big adva gage. I will try it out to see how it goes. It goes without saying that backups of this data will be important!

The Docker applications that I will run initially:

1. Portainer
2. Heimdall
3. NextCloud
4. Paperless NG
5. A TBD image library manager

Most of the containers on this list have been attempted without succes from the TrueNAS application catalog. The exception is Heimdall which I find very helpful.

Hardware configuration

The shopping list for this build includes the following components:

QTY	Item Description	$
1	Dell PowerEdge R530 Server 32GRAM Dual E5-2620 v3	120	eBay
1	Dell Perc HBA H330 Mini Raid controller	15	eBay
2	A-Tech 32GB 4Rx4 PC4-2133 LRDIMM RAM	80	eBay
5	Seagate 4TB SAS HDD	75	eBay
1	HUS7260ALS210 SAS HDD	15	eBay
1	WD 2TB Blue SATA SSD	120	Amazon
7	Dell 3.5” SATA/SAS HDD Caddy R530	50	eBay
1	ATA SSD caddy adapter replace CD drive	5	on hand
-	Total project equipment cost	480
1	Sold R530 Bezzle	-10	eBay
1	Sold PERC H730P	-30	eBay
-	Final project cost	440

I did not need the PERC H730 RAID controller and the R530 bezel that came with the R530. They got sole on eBay to resuce the cost of the build a little. Also, this build needs additional equipment to add a system drive for TrueNAS, right now this is a virtual drive which is not supportted by TrueNAS.

The R530 chassis is configured in the following way:

1. 6x 4TB HDD RAIDZ1 ZFS pool for general NAS storage PCI passthrough of the H330 to TrueNAS VM.
2. 1x 2TB SDD in SATA CD caddy for Proxmox lvm storage.

Update internal firmware

Run the Dell System Update Utility (SUU) for the Poweredge R530.

First install MS Windows 10. I used one of the HDDs as tempoary Windows Installation.

Then go to www.dell.com -> Support -> Drivers and downloads Identify your product -> Poweredge R720 (Next) Keyword -> SUU Download: DELL Server Update Utility, Windows 64 bit Format, v.22.11.00

The Dell System Update Utility first takes an inventory of existing firmware. Select from the discovered inventory all the firmware components and ignore the windows driver updates. It takes a while to collect inventory and perform the update.

The SUU did an update to BIOS 2.15. There was an update to version 2.19 that had to be done manually.

Install Proxmox

Follow the instructions from Derek Seaman's Tech Blog: Quick Start Guide.

The scripts from Derek made very quick work of configuring Proxmox. (Thanks Derek! Very well done indeed!) For this build, follow the steps for the Proxmox installation and configuration only. Stop before the Home Assistant install part of the description.

Next enable IOMMU passthrough in the BIOS.

See PCI passthrough in Proxmox documentation for details. The following command verifies that the BIOS setus is correct:

 dmesg | grep -e DMAR -e IOMMU | grep  2apic
[    0.629383] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.630036] DMAR-IR: Enabled IRQ remapping in x2apic mode

Last thing, I found a nifty method to enable dark mode on Proxmox. The github page for the script

Install TrueNAS CORE

I found a handy guide for installing TrueNAS Core.

Page 6 is the relevant page for me. The information that saved lots of trouble is setting hardware to qx35, and check qemu agent on the system page during the setup.

Proxmox does not assign a serial number to virtual disks. I'm using a virtual disk for the system disk for now and TrueNAS rejects pool create operations when disks with duplicate serial numbers are found. Two disks with blank serial numbers are considered duplicate. A serial number must be manually assigned to a virtual disk. Proxmox documentation describes where to find the vm.conf file.

This is a work around for now. Must provide hardware access to a disk to finalize this part of the build.

Install Plex in LXC container

First create Plex media NFS share TrueNAS core

I found a the following blog Plex install on Proxmox LXC

Modify the procedure from as follows:

• Set ipv6 to static because there is no DHCP derver in my network.
• nfs gave UID/GID in plex container as 3000
• Create user nfsadmin with GIT=3000
  • groupadd -g 3000 nfsadmin
• assign the plex user to that group
  • usermod -a -G nfsadmin plex

Install Docker LXC container

Installing docker in LXC

• 100G disk size and 8 CPUs 2048 K ram

After docker install creat an admin user

   mkdir /home/admin
   useradd --home /home/admin admin
   usermod -aG sudo admin
   password admin
   chown admin:admin /home/admin

Now allow admin to run docker without sudo

usermod -aG docker admin
su admin
docker run hello-world

change the default shell to bash

chsh

after enter password enter /bin/bash Create

now convert the docker run command to a docker-compose.yml https://www.composerize.com/

Edit the result to use local data volume for container /home/admin/portiner/data:/data

Power Consumption Measurements

At Idle with all drives powered up 0.73A => 88 W Running rsync copy at 900Mbps lan 1.2A to 2A => 144W typ peak 240W peak

The system runs 24 x 7 and moistly runs less than 90W.