Add README.md
This commit is contained in:
commit
3d41cf0f70
1 changed files with 158 additions and 0 deletions
158
README.md
Normal file
158
README.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
# Promxox PVE w/ TrueNAS Core VM
|
||||
|
||||
I've been running my homelab for a few months now with a Dell R720 enterprise server running TrueNAS Scale as the primary NAS and as a virtualization server. My backup NAS is a Dell Optiplex 5090 MT also running TrueNAS Scale.
|
||||
The R720 runs Plex Media Server and Heimdall as applications in TrueNAS Scale.
|
||||
|
||||
The Optiplex runs a Windows VM under TrueNAS for BlueIris. This machine includes a single 10TB HDD serving as video storage and as a replication target for the primary NAS.
|
||||
|
||||
I would really like to use the TrueNAS application library, however support is limited and many of the applications that I've tried simply do not run or crash. I fully acknowledge that this result may be my own limited ability. But several TrueNAS blog posts talk about how the TrueNAS application catalog is not well maintained. I will move to a different solution for now.
|
||||
|
||||
The subject of this gist is my next Home Lab project: build a replacement virtualization server that also doubles as a NAS. The existing TruenNAS server shall become the backup and will not support any virtulization. Ultamately, the R720 will be replaced with something that is lower power to serve as the backup NAS.
|
||||
|
||||
The future of the BlueIris host machine is still an open question. I'm leaning to installing Proxmox on this machine too. I've read that a Proxmox ZFS pool can be the target of a TrueNAS remote replication. More on this topic later.
|
||||
|
||||
## Requirements for this build
|
||||
1. Repurpose enterprise server hardware with 8 HDD bays and ECC memory
|
||||
2. Proxmox as the virtualization server
|
||||
3. TrueNAS core running in a Proxmox VM
|
||||
|
||||
Historically I've run TrueNAS core as my NAS it has been reliable for me. Also, it is the recomended version for production deployments. So for now my experiment with TrueNAS Scale is over. I will use TrueNAS and Proxmox for what each does best. In this build, TrueNAS shall operate only as a NAS. None of its applications shall be deployed and there will not be any VMs running with it.
|
||||
|
||||
Applications that will run as Proxmox LXC containers:
|
||||
1. Plex Media Server
|
||||
2. Docker
|
||||
|
||||
Running Docker in an LXC container is not recomended by the Proxmox support team. However. I've been reading many blog posts from users that do this successfully. The advantage of the LXC container is mamory allocation; the VM is assigned a fixed allocation and the LXC container is dynamically assigned as needed. This dynamic allocation seems like a big adva gage. I will try it out to see how it goes. It goes without saying that backups of this data will be important!
|
||||
|
||||
The Docker applications that I will run initially:
|
||||
1. Portainer
|
||||
2. Heimdall
|
||||
3. NextCloud
|
||||
4. Paperless NG
|
||||
5. A TBD image library manager
|
||||
|
||||
Most of the containers on this list have been attempted without succes from the TrueNAS application catalog. The exception is Heimdall which I find very helpful.
|
||||
|
||||
## Hardware configuration
|
||||
|
||||
The shopping list for this build includes the following components:
|
||||
|
||||
| QTY | Item Description | $ | |
|
||||
|-----|--------------------------------------------------|----:|---------|
|
||||
| 1 | Dell PowerEdge R530 Server 32GRAM Dual E5-2620 v3| 120 | eBay |
|
||||
| 1 | Dell Perc HBA H330 Mini Raid controller | 15 | eBay |
|
||||
| 2 | A-Tech 32GB 4Rx4 PC4-2133 LRDIMM RAM | 80 | eBay |
|
||||
| 5 | Seagate 4TB SAS HDD | 75 | eBay |
|
||||
| 1 | HUS7260ALS210 SAS HDD | 15 | eBay |
|
||||
| 1 | WD 2TB Blue SATA SSD | 120 | Amazon |
|
||||
| 7 | Dell 3.5” SATA/SAS HDD Caddy R530 | 50 | eBay |
|
||||
| 1 | ATA SSD caddy adapter replace CD drive | 5 | on hand |
|
||||
| - | Total project equipment cost | 480 | |
|
||||
| 1 | Sold R530 Bezzle | -10 | eBay |
|
||||
| 1 | Sold PERC H730P | -30 | eBay |
|
||||
| - | Final project cost | 440 |
|
||||
|
||||
I did not need the PERC H730 RAID controller and the R530 bezel that came with the R530. They got sole on eBay to resuce the cost of the build a little. Also, this build needs additional equipment to add a system drive for TrueNAS, right now this is a virtual drive which is not supportted by TrueNAS.
|
||||
|
||||
The R530 chassis is configured in the following way:
|
||||
1. 6x 4TB HDD RAIDZ1 ZFS pool for general NAS storage PCI passthrough of the H330 to TrueNAS VM.
|
||||
2. 1x 2TB SDD in SATA CD caddy for Proxmox lvm storage.
|
||||
|
||||
## Update internal firmware
|
||||
|
||||
Run the Dell System Update Utility (SUU) for the Poweredge R530.
|
||||
|
||||
First install MS Windows 10. I used one of the HDDs as tempoary [Windows Installation](https://www.microsoft.com/en-us/software-download/windows10).
|
||||
|
||||
Then go to www.dell.com -> Support -> Drivers and downloads
|
||||
Identify your product -> Poweredge R720 (Next)
|
||||
Keyword -> SUU
|
||||
Download: DELL Server Update Utility, Windows 64 bit Format, v.22.11.00
|
||||
|
||||
The Dell System Update Utility first takes an inventory of existing firmware. Select from the discovered inventory all the firmware components and ignore the windows driver updates. It takes a while to collect inventory and perform the update.
|
||||
|
||||
The SUU did an update to BIOS 2.15. There was an update to version 2.19 that had to be done manually.
|
||||
|
||||
## Install Proxmox
|
||||
Follow the instructions from Derek Seaman's Tech Blog: [Quick Start Guide](https://www.derekseaman.com/2023/06/home-assistant-proxmox-ve-8-0-quick-start-guide.html).
|
||||
|
||||
The scripts from Derek made very quick work of configuring Proxmox. (Thanks Derek! Very well done indeed!)
|
||||
For this build, follow the steps for the Proxmox installation and configuration only. Stop before the Home Assistant install part of the description.
|
||||
|
||||
Next enable IOMMU passthrough in the BIOS.
|
||||
|
||||
See [PCI passthrough](https://pve.proxmox.com/wiki/PCI_Passthrough) in Proxmox documentation for details.
|
||||
The following command verifies that the BIOS setus is correct:
|
||||
```
|
||||
dmesg | grep -e DMAR -e IOMMU | grep 2apic
|
||||
[ 0.629383] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
|
||||
[ 0.630036] DMAR-IR: Enabled IRQ remapping in x2apic mode
|
||||
```
|
||||
Last thing, I found a nifty method to [enable dark mode on Proxmox](https://computingforgeeks.com/customize-proxmox-ve-web-ui-with-dark-theme/). [The github page for the script](https://github.com/Weilbyte/PVEDiscordDark/blob/master/README.md)
|
||||
|
||||
## Install TrueNAS CORE
|
||||
I found a handy guide for [installing TrueNAS Core](https://www.snakeoil-os.net/news/under-the-sun/complete-guide-to-truenas-core-virtualisation-under-proxmox?page=6).
|
||||
|
||||
Page 6 is the relevant page for me. The information that saved lots of trouble is setting hardware to qx35, and check qemu agent on the system page during the setup.
|
||||
|
||||
Proxmox does not assign a serial number to virtual disks. I'm using a virtual disk for the system disk for now and TrueNAS rejects pool create operations when disks with duplicate serial numbers are found. Two disks with blank serial numbers are considered duplicate. [A serial number must be manually assigned to a virtual disk](https://forum.proxmox.com/threads/setting-hard-drive-serial-number.20954/). Proxmox documentation describes [where to find the vm.conf file](https://pve.proxmox.com/wiki/Manual:_qm.conf).
|
||||
|
||||
This is a work around for now. Must provide hardware access to a disk to finalize this part of the build.
|
||||
|
||||
## Install Plex in LXC container
|
||||
|
||||
First [create Plex media NFS share TrueNAS core](https://www.truenas.com/docs/core/coretutorials/sharing/nfsshare/)
|
||||
|
||||
I found a the following blog [Plex install on Proxmox LXC](https://www.wundertech.net/how-to-install-plex-on-proxmox/)
|
||||
|
||||
Modify the procedure from as follows:
|
||||
- Set ipv6 to static because there is no DHCP derver in my network.
|
||||
- nfs gave UID/GID in plex container as 3000
|
||||
- Create user nfsadmin with GIT=3000
|
||||
- groupadd -g 3000 nfsadmin
|
||||
- assign the plex user to that group
|
||||
- usermod -a -G nfsadmin plex
|
||||
|
||||
## Install Docker LXC container
|
||||
|
||||
[Installing docker in LXC](https://www.wundertech.net/how-to-set-up-docker-containers-in-proxmox/)
|
||||
|
||||
- 100G disk size and 8 CPUs 2048 K ram
|
||||
|
||||
After docker install creat an admin user
|
||||
```
|
||||
mkdir /home/admin
|
||||
useradd --home /home/admin admin
|
||||
usermod -aG sudo admin
|
||||
password admin
|
||||
chown admin:admin /home/admin
|
||||
```
|
||||
|
||||
Now allow admin to run docker without sudo
|
||||
```
|
||||
usermod -aG docker admin
|
||||
su admin
|
||||
docker run hello-world
|
||||
```
|
||||
change the default shell to bash
|
||||
```
|
||||
chsh
|
||||
```
|
||||
after enter password enter /bin/bash
|
||||
Create
|
||||
|
||||
|
||||
now convert the docker run command to a docker-compose.yml
|
||||
https://www.composerize.com/
|
||||
|
||||
Edit the result to use local data volume for container
|
||||
/home/admin/portiner/data:/data
|
||||
|
||||
|
||||
|
||||
## Power Consumption Measurements
|
||||
|
||||
At Idle with all drives powered up 0.73A => 88 W
|
||||
Running rsync copy at 900Mbps lan 1.2A to 2A => 144W typ peak 240W peak
|
||||
|
||||
The system runs 24 x 7 and moistly runs less than 90W.
|
||||
Loading…
Reference in a new issue