From 8902c2d427ba2f81e96838471606147b2514412e Mon Sep 17 00:00:00 2001 From: hhf Date: Tue, 26 Nov 2024 16:16:53 +0530 Subject: [PATCH] Add README.md --- README.md | 171 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 README.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..bd6896d --- /dev/null +++ b/README.md @@ -0,0 +1,171 @@ +
+ + + + +### My Homelab Repository :snowflake: + +_... automated via [Flux](https://fluxcd.io), [Renovate](https://github.com/renovatebot/renovate) and [GitHub Actions](https://github.com/features/actions)_ 🤖 + +
+--- + +## Overview + +This is a monorepository is for my home kubernetes clusters. +I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like [Terraform](https://www.terraform.io/), [Kubernetes](https://kubernetes.io/), [Flux](https://github.com/fluxcd/flux2), [Renovate](https://github.com/renovatebot/renovate), and [GitHub Actions](https://github.com/features/actions). + +The purpose here is to learn k8s, while practicing Gitops. + +--- + +## ⛵ Kubernetes + +There is a template over at [onedr0p/cluster-template](https://github.com/onedr0p/cluster-template) if you want to try and follow along with some of the practices I use here. + +### Installation + +My clusters run [talos linux](https://www.talos.dev) immutable kubernetes OS. This is a semi-hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate NAS server running Unraid withg NFS/SMB shares, bulk file storage and backups. + +### Core Components + +- [actions-runner-controller](https://github.com/actions/actions-runner-controller): self-hosted Github runners +- [cilium](https://github.com/cilium/cilium): internal Kubernetes networking plugin +- [cert-manager](https://cert-manager.io/docs/): creates SSL certificates for services in my cluster +- [external-dns](https://github.com/kubernetes-sigs/external-dns): automatically syncs DNS records from my cluster ingresses to a DNS provider +- [external-secrets](https://github.com/external-secrets/external-secrets/): managed Kubernetes secrets using [Bitwarden](https://bitwarden.com/). +- [ingress-nginx](https://github.com/kubernetes/ingress-nginx/): ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer +- [longhorn](https://longhorn.io/): Cloud native distributed block storage for Kubernetes +- [rook-ceph](https://rook.io/): Cloud native distributed block storage for Kubernetes +- [sops](https://toolkit.fluxcd.io/guides/mozilla-sops/): managed secrets for Kubernetes, Ansible, and Terraform which are committed to Git +- [spegel](https://github.com/XenitAB/spegel): stateless cluster local OCI registry mirror +- [tf-controller](https://github.com/weaveworks/tf-controller): additional Flux component used to run Terraform from within a Kubernetes cluster. +- [volsync](https://github.com/backube/volsync): backup and recovery of persistent volume claims + +### GitOps + +[Flux](https://github.com/fluxcd/flux2) watches the clusters in my [kubernetes](./kubernetes/) folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository. + +The way Flux works for me here is it will recursively search the `kubernetes/${cluster}/apps` folder until it finds the most top level `kustomization.yaml` per directory and then apply all the resources listed in it. That aforementioned `kustomization.yaml` will generally only have a namespace resource and one or many Flux kustomizations. Those Flux kustomizations will generally have a `HelmRelease` or other resources related to the application underneath it which will be applied. + +[Renovate](https://github.com/renovatebot/renovate) watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster. + +### Directories + +This Git repository contains the following directories under [Kubernetes](./kubernetes/). + +```sh +📁 kubernetes +├── 📁 main # main cluster +│ ├── 📁 apps # applications +│ ├── 📁 bootstrap # bootstrap procedures +│ ├── 📁 flux # core flux configuration +│ └── 📁 talos # talos configuration +├── 📁 shared # shared cluster resources +└── 📁 utility # utility cluster + ├── 📁 apps # applications + ├── 📁 bootstrap # bootstrap procedures + ├── 📁 flux # core flux configuration + └── 📁 talos # talos configuration +``` + +### Networking + +
+ Click to see a high-level network diagram + + dns +
+ +--- + +## ☁️ Cloud Dependencies + +While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not. + +The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like [HCVault](https://www.vaultproject.io/), [Vaultwarden](https://github.com/dani-garcia/vaultwarden), [ntfy](https://ntfy.sh/), and [Gatus](https://gatus.io/). However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in. + +| Service | Use | Cost | +|---------------------------------------------|-------------------------------------------------------------------|----------------| +| [Bitwarden](https://bitwarden.com/) | Secrets with [External Secrets](https://external-secrets.io/) | ~$10/yr | +| [Cloudflare](https://www.cloudflare.com/) | Domain and R2 | ~$30/yr | +| [GitHub](https://github.com/) | Hosting this repository and continuous integration/deployments | Free | +| [Healthchecks.io](https://healthchecks.io/) | Monitoring internet connectivity and external facing applications | Free | +| | | Total: ~$5/mo | + +--- + +## 🌐 DNS + +### Home DNS + +In my cluster `external-dns` is deployed with the `RFC2136` provider which syncs DNS records to `unifi` via [external-dns-unifi-webhook](https://github.com/kashalls/external-dns-unifi-webhook). + +### Public DNS + +Outside the `external-dns` instance mentioned above another instance is deployed in my cluster and configured to sync DNS records to [Cloudflare](https://www.cloudflare.com/). The only ingress this `external-dns` instance looks at to gather DNS records to put in `Cloudflare` are ones that have an ingress class name of `external` and contain an ingress annotation `external-dns.alpha.kubernetes.io/target`. + +--- + +## 🔧 Hardware + +### Main Kubernetes Cluster + +| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose | +|-------|--------------|-----------|-----------|-----------|------|-------|-------------------| +| Ayaka | Dell 7080mff | i5-10500T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane | +| Eula | Dell 7080mff | i7-10700T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane | +| Ganyu | Dell 3080mff | i5-10500T | 240GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane | +| HuTao | Dell 3080mff | i5-10500T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s worker | +| Navia | Dell 3080mff | i5-10500T | 256GB SSD | 1TB NVME | 64GB | Talos | k8s worker | +| Yelan | Dell 3080mff | i5-10500T | 240GB SSD | 1TB NVME | 64GB | Talos | k8s worker | + +Total CPU: 76 threads +Total RAM: 384GB + +### Utility Kubernetes Cluster + +| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose | +|----------|------------|---------------|-----------|------------|------|-------|-------------------| +| Celestia | Bosgame P1 | Ryzen 7 5700U | 480GB SSD | 512GB NVME | 32GB | Talos | k8s control-plane | + +Total CPU: 16 threads +Total RAM: 32GB + +### Supporting Hardware + +| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose | +|--------|----------------|---------------|------------|------------|-------|--------------|----------------| +| NAS | HP z820 | E5-2680v2 | 32GB USB | 500GB NVMe | 128GB | Unraid | NAS/NFS/Backup | +| DAS | Lenovo SA120 | - | - | 56TB | - | - | ZFS - Raidz2 | +| PiKVM | Raspberry Pi4 | Cortex A72 | 64GB mSD | - | 4GB | PiKVM (Arch) | KVM | + +### Networking/UPS Hardware + +| Device | Purpose | +|-----------------------------|------------------| +| Unifi UDM-SE | Network - Router | +| USW-Pro-24-POE | Network - Switch | +| Back-UPS 600 | Network - UPS | +| Unifi USW-Enterprise-24-PoE | Server - Switch | +| Tripp Lite 1500 | Server - UPS | + +--- + +## 🤝 Thanks + +Big shout out to original [cluster-template](https://github.com/onedr0p/cluster-template), and the [Home Operations](https://discord.gg/home-operations) Discord community. + +Be sure to check out [kubesearch.dev](https://kubesearch.dev/) for ideas on how to deploy applications or get ideas on what you may deploy. + +--- + +## 📜 Changelog + + + +--- + +## 🔏 License + +See [LICENSE](./LICENSE) \ No newline at end of file