Add README.md
This commit is contained in:
commit
8902c2d427
1 changed files with 171 additions and 0 deletions
171
README.md
Normal file
171
README.md
Normal file
|
|
@ -0,0 +1,171 @@
|
||||||
|
<div align="center">
|
||||||
|
|
||||||
|
<img src="https://raw.githubusercontent.com/joryirving/home-ops/main/docs/src/assets/icons/kah-logo.png" align="center" width="144px" height="144px"/>
|
||||||
|
|
||||||
|
|
||||||
|
### My Homelab Repository :snowflake:
|
||||||
|
|
||||||
|
_... automated via [Flux](https://fluxcd.io), [Renovate](https://github.com/renovatebot/renovate) and [GitHub Actions](https://github.com/features/actions)_ 🤖
|
||||||
|
|
||||||
|
</div>
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This is a monorepository is for my home kubernetes clusters.
|
||||||
|
I try to adhere to Infrastructure as Code (IaC) and GitOps practices using tools like [Terraform](https://www.terraform.io/), [Kubernetes](https://kubernetes.io/), [Flux](https://github.com/fluxcd/flux2), [Renovate](https://github.com/renovatebot/renovate), and [GitHub Actions](https://github.com/features/actions).
|
||||||
|
|
||||||
|
The purpose here is to learn k8s, while practicing Gitops.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⛵ Kubernetes
|
||||||
|
|
||||||
|
There is a template over at [onedr0p/cluster-template](https://github.com/onedr0p/cluster-template) if you want to try and follow along with some of the practices I use here.
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
My clusters run [talos linux](https://www.talos.dev) immutable kubernetes OS. This is a semi-hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate NAS server running Unraid withg NFS/SMB shares, bulk file storage and backups.
|
||||||
|
|
||||||
|
### Core Components
|
||||||
|
|
||||||
|
- [actions-runner-controller](https://github.com/actions/actions-runner-controller): self-hosted Github runners
|
||||||
|
- [cilium](https://github.com/cilium/cilium): internal Kubernetes networking plugin
|
||||||
|
- [cert-manager](https://cert-manager.io/docs/): creates SSL certificates for services in my cluster
|
||||||
|
- [external-dns](https://github.com/kubernetes-sigs/external-dns): automatically syncs DNS records from my cluster ingresses to a DNS provider
|
||||||
|
- [external-secrets](https://github.com/external-secrets/external-secrets/): managed Kubernetes secrets using [Bitwarden](https://bitwarden.com/).
|
||||||
|
- [ingress-nginx](https://github.com/kubernetes/ingress-nginx/): ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer
|
||||||
|
- [longhorn](https://longhorn.io/): Cloud native distributed block storage for Kubernetes
|
||||||
|
- [rook-ceph](https://rook.io/): Cloud native distributed block storage for Kubernetes
|
||||||
|
- [sops](https://toolkit.fluxcd.io/guides/mozilla-sops/): managed secrets for Kubernetes, Ansible, and Terraform which are committed to Git
|
||||||
|
- [spegel](https://github.com/XenitAB/spegel): stateless cluster local OCI registry mirror
|
||||||
|
- [tf-controller](https://github.com/weaveworks/tf-controller): additional Flux component used to run Terraform from within a Kubernetes cluster.
|
||||||
|
- [volsync](https://github.com/backube/volsync): backup and recovery of persistent volume claims
|
||||||
|
|
||||||
|
### GitOps
|
||||||
|
|
||||||
|
[Flux](https://github.com/fluxcd/flux2) watches the clusters in my [kubernetes](./kubernetes/) folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.
|
||||||
|
|
||||||
|
The way Flux works for me here is it will recursively search the `kubernetes/${cluster}/apps` folder until it finds the most top level `kustomization.yaml` per directory and then apply all the resources listed in it. That aforementioned `kustomization.yaml` will generally only have a namespace resource and one or many Flux kustomizations. Those Flux kustomizations will generally have a `HelmRelease` or other resources related to the application underneath it which will be applied.
|
||||||
|
|
||||||
|
[Renovate](https://github.com/renovatebot/renovate) watches my **entire** repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
|
||||||
|
|
||||||
|
### Directories
|
||||||
|
|
||||||
|
This Git repository contains the following directories under [Kubernetes](./kubernetes/).
|
||||||
|
|
||||||
|
```sh
|
||||||
|
📁 kubernetes
|
||||||
|
├── 📁 main # main cluster
|
||||||
|
│ ├── 📁 apps # applications
|
||||||
|
│ ├── 📁 bootstrap # bootstrap procedures
|
||||||
|
│ ├── 📁 flux # core flux configuration
|
||||||
|
│ └── 📁 talos # talos configuration
|
||||||
|
├── 📁 shared # shared cluster resources
|
||||||
|
└── 📁 utility # utility cluster
|
||||||
|
├── 📁 apps # applications
|
||||||
|
├── 📁 bootstrap # bootstrap procedures
|
||||||
|
├── 📁 flux # core flux configuration
|
||||||
|
└── 📁 talos # talos configuration
|
||||||
|
```
|
||||||
|
|
||||||
|
### Networking
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>Click to see a high-level network diagram</summary>
|
||||||
|
|
||||||
|
<img src="https://raw.githubusercontent.com/joryirving/home-ops/main/docs/src/assets/network-topology.png" align="center" alt="dns"/>
|
||||||
|
</details>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ☁️ Cloud Dependencies
|
||||||
|
|
||||||
|
While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.
|
||||||
|
|
||||||
|
The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like [HCVault](https://www.vaultproject.io/), [Vaultwarden](https://github.com/dani-garcia/vaultwarden), [ntfy](https://ntfy.sh/), and [Gatus](https://gatus.io/). However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in.
|
||||||
|
|
||||||
|
| Service | Use | Cost |
|
||||||
|
|---------------------------------------------|-------------------------------------------------------------------|----------------|
|
||||||
|
| [Bitwarden](https://bitwarden.com/) | Secrets with [External Secrets](https://external-secrets.io/) | ~$10/yr |
|
||||||
|
| [Cloudflare](https://www.cloudflare.com/) | Domain and R2 | ~$30/yr |
|
||||||
|
| [GitHub](https://github.com/) | Hosting this repository and continuous integration/deployments | Free |
|
||||||
|
| [Healthchecks.io](https://healthchecks.io/) | Monitoring internet connectivity and external facing applications | Free |
|
||||||
|
| | | Total: ~$5/mo |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🌐 DNS
|
||||||
|
|
||||||
|
### Home DNS
|
||||||
|
|
||||||
|
In my cluster `external-dns` is deployed with the `RFC2136` provider which syncs DNS records to `unifi` via [external-dns-unifi-webhook](https://github.com/kashalls/external-dns-unifi-webhook).
|
||||||
|
|
||||||
|
### Public DNS
|
||||||
|
|
||||||
|
Outside the `external-dns` instance mentioned above another instance is deployed in my cluster and configured to sync DNS records to [Cloudflare](https://www.cloudflare.com/). The only ingress this `external-dns` instance looks at to gather DNS records to put in `Cloudflare` are ones that have an ingress class name of `external` and contain an ingress annotation `external-dns.alpha.kubernetes.io/target`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔧 Hardware
|
||||||
|
|
||||||
|
### Main Kubernetes Cluster
|
||||||
|
|
||||||
|
| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose |
|
||||||
|
|-------|--------------|-----------|-----------|-----------|------|-------|-------------------|
|
||||||
|
| Ayaka | Dell 7080mff | i5-10500T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane |
|
||||||
|
| Eula | Dell 7080mff | i7-10700T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane |
|
||||||
|
| Ganyu | Dell 3080mff | i5-10500T | 240GB SSD | 1TB NVME | 64GB | Talos | k8s control-plane |
|
||||||
|
| HuTao | Dell 3080mff | i5-10500T | 480GB SSD | 1TB NVME | 64GB | Talos | k8s worker |
|
||||||
|
| Navia | Dell 3080mff | i5-10500T | 256GB SSD | 1TB NVME | 64GB | Talos | k8s worker |
|
||||||
|
| Yelan | Dell 3080mff | i5-10500T | 240GB SSD | 1TB NVME | 64GB | Talos | k8s worker |
|
||||||
|
|
||||||
|
Total CPU: 76 threads
|
||||||
|
Total RAM: 384GB
|
||||||
|
|
||||||
|
### Utility Kubernetes Cluster
|
||||||
|
|
||||||
|
| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose |
|
||||||
|
|----------|------------|---------------|-----------|------------|------|-------|-------------------|
|
||||||
|
| Celestia | Bosgame P1 | Ryzen 7 5700U | 480GB SSD | 512GB NVME | 32GB | Talos | k8s control-plane |
|
||||||
|
|
||||||
|
Total CPU: 16 threads
|
||||||
|
Total RAM: 32GB
|
||||||
|
|
||||||
|
### Supporting Hardware
|
||||||
|
|
||||||
|
| Name | Device | CPU | OS Disk | Data Disk | RAM | OS | Purpose |
|
||||||
|
|--------|----------------|---------------|------------|------------|-------|--------------|----------------|
|
||||||
|
| NAS | HP z820 | E5-2680v2 | 32GB USB | 500GB NVMe | 128GB | Unraid | NAS/NFS/Backup |
|
||||||
|
| DAS | Lenovo SA120 | - | - | 56TB | - | - | ZFS - Raidz2 |
|
||||||
|
| PiKVM | Raspberry Pi4 | Cortex A72 | 64GB mSD | - | 4GB | PiKVM (Arch) | KVM |
|
||||||
|
|
||||||
|
### Networking/UPS Hardware
|
||||||
|
|
||||||
|
| Device | Purpose |
|
||||||
|
|-----------------------------|------------------|
|
||||||
|
| Unifi UDM-SE | Network - Router |
|
||||||
|
| USW-Pro-24-POE | Network - Switch |
|
||||||
|
| Back-UPS 600 | Network - UPS |
|
||||||
|
| Unifi USW-Enterprise-24-PoE | Server - Switch |
|
||||||
|
| Tripp Lite 1500 | Server - UPS |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🤝 Thanks
|
||||||
|
|
||||||
|
Big shout out to original [cluster-template](https://github.com/onedr0p/cluster-template), and the [Home Operations](https://discord.gg/home-operations) Discord community.
|
||||||
|
|
||||||
|
Be sure to check out [kubesearch.dev](https://kubesearch.dev/) for ideas on how to deploy applications or get ideas on what you may deploy.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📜 Changelog
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔏 License
|
||||||
|
|
||||||
|
See [LICENSE](./LICENSE)
|
||||||
Loading…
Reference in a new issue