k3s cluster backed by Flux (archived due to colo costs)
The purpose here is to showcase how you can deploy an entire Kubernetes cluster and show it off to the world using the GitOps tool Flux. When completed, your Git repository will be driving the state of your Kubernetes cluster. In addition with the help of the Ansible, Terraform and Flux SOPS integrations you'll be able to commit Age encrypted secrets to your public repo.
- Repository structure
- Lets go!
- Post installation
- What's next
The following components will be installed in your k3s cluster by default. Most are only included to get a minimum viable cluster up and running.
- flux - GitOps operator for managing Kubernetes clusters from a Git repository
- metallb - Load balancer for Kubernetes services
- cert-manager - Operator to request SSL certificates and store them as Kubernetes resources
- external-dns - Operator to publish DNS records to Cloudflare (and other providers) based on Kubernetes ingresses
- ingress-nginx - Kubernetes ingress controller used for a HTTP reverse proxy of Kubernetes ingresses
- local-path-provisioner - provision persistent local storage with Kubernetes
For provisioning the following tools will be used:
- Fedora 36 Server - Universal operating system that supports running all kinds of home related workloads in Kubernetes and has a faster release cycle
- Ansible - Provision Fedora Server and install k3s
Note: This template has not been tested on cloud providers like AWS EC2, Hetzner, Scaleway etc... Those cloud offerings probably have a better way of provsioning a Kubernetes cluster and it's advisable to use those instead of the Ansible playbooks included here. This repository can still be tweaked for the GitOps/Flux portion if there's a cluster working in one those environments.
📚 Reading material
- One or more nodes with a fresh install of Fedora Server 36.
- These nodes can be ARM64/AMD64 bare metal or VMs.
- An odd number of control plane nodes, greater than or equal to 3 is required if deploying more than one control plane node.
- A Cloudflare account with a domain, this will be managed by Terraform and external-dns. You can register new domains directly thru Cloudflare.
- Some experience in debugging problems and a positive attitude ;)
📍 It is recommended to have 3 master nodes for a highly available control plane.
🔧 Workstation Tools
Install the most recent versions of the following CLI tools on your workstation, if you are using Homebrew on MacOS or Linux skip to steps 3 and 4.
This guide heavily relies on go-task as a framework for setting things up. It is advised to learn and understand the commands it is running under the hood.
Install go-task via Brew
brew install go-task/tap/go-task
Install workstation dependencies via Brew
It is advisable to install pre-commit and the pre-commit hooks that come with this repository. sops-pre-commit will check to make sure you are not committing non-encrypted Kubernetes secrets to your repository.
Update Pre-Commit, though it will occasionally make mistakes, so verify its results.
📂 Repository structure
The Git repository contains the following directories under
cluster and are ordered below by how Flux will apply them.
📁 cluster # k8s cluster defined as code ├─📁 flux # flux, gitops operator, loaded before everything ├─📁 crds # custom resources, loaded before 📁 core and 📁 apps ├─📁 charts # helm repos, loaded before 📁 core and 📁 apps ├─📁 config # cluster config, loaded before 📁 core and 📁 apps ├─📁 core # crucial apps, namespaced dir tree, loaded before 📁 apps └─📁 apps # regular apps, namespaced dir tree, loaded last
Big shout out to all the authors and contributors to the projects that we are using in this repository.