Field note status: this article documents the current architecture and the target design. The worker redistribution is planned, not presented as already completed.
I had three k3s control-plane nodes and four worker nodes. On paper, that looked reasonably redundant for a small on-premises cluster.
Then I drew the physical layout.
Control plane
├── control-plane-01 — physical mini PC
├── control-plane-02 — physical mini PC
└── control-plane-03 — physical mini PC
Proxmox host A
└── mostly infrastructure VMs
Proxmox host B
├── k3s-worker-01
├── k3s-worker-02
├── k3s-worker-03
└── k3s-worker-04
The control plane was distributed across three separate machines, but every worker VM lived on the same Proxmox host. If that host failed, Kubernetes could still have a healthy control plane while losing practically every application workload.
Four worker nodes did not give me four independent failure domains. They gave me four virtual machines inside one physical failure domain.
HA starts below Kubernetes
Kubernetes can reschedule a Pod after a node failure, but only if another usable node still exists. It cannot move a workload away from a failed hypervisor when every worker belongs to that hypervisor.
This distinction is easy to miss in a virtualized homelab or small business environment:
- Node redundancy means having multiple Kubernetes nodes.
- Failure-domain redundancy means those nodes do not all depend on the same physical host, power path, network path or storage system.
The scheduler sees four workers. The person designing the platform must also see the Proxmox hosts underneath them.
This is why the next step in my migration plan is not moving another application to k3s. It is fixing the worker placement first.
The target layout
The least disruptive option is to add two worker VMs to the other Proxmox host instead of immediately moving existing workers.
Control plane
├── control-plane-01 — physical mini PC
├── control-plane-02 — physical mini PC
└── control-plane-03 — physical mini PC
Proxmox host A
├── k3s-worker-05
└── k3s-worker-06
Proxmox host B
├── k3s-worker-01
├── k3s-worker-02
├── k3s-worker-03
└── k3s-worker-04
This temporarily creates six workers, but it avoids disturbing the running cluster while the second physical failure domain is introduced. Once the new workers are joined, labelled and tested, I can decide whether six workers are useful or whether some of the original VMs should be drained and retired.
The final layout may be two workers on each Proxmox host. The important part is not the exact count. The important part is that losing either virtualization host must leave enough worker capacity to run the critical services.
Teach the scheduler about the physical topology
Kubernetes does not automatically know that two virtual machines share the same hypervisor. I need to describe that topology with node labels.
An anonymized version of the planned labels looks like this:
kubectl label node k3s-worker-01 topology.kubernetes.io/zone=proxmox-b
kubectl label node k3s-worker-02 topology.kubernetes.io/zone=proxmox-b
kubectl label node k3s-worker-03 topology.kubernetes.io/zone=proxmox-b
kubectl label node k3s-worker-04 topology.kubernetes.io/zone=proxmox-b
kubectl label node k3s-worker-05 topology.kubernetes.io/zone=proxmox-a
kubectl label node k3s-worker-06 topology.kubernetes.io/zone=proxmox-a
The zone names are logical failure domains. They do not need to be cloud availability zones. In this environment, each value represents one physical Proxmox host.
Important multi-replica workloads can then use Pod topology spread constraints so their replicas are distributed across those domains.
spec:
replicas: 2
template:
metadata:
labels:
app: example-service
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: example-service
This example says that replicas of the same application should not be concentrated in one Proxmox failure domain. It is a stronger guarantee than simply hoping the scheduler will choose different nodes.
I will not apply this blindly to every Pod. A single-replica internal tool cannot become highly available through a scheduling rule, and a hard DoNotSchedule constraint can leave a Pod Pending when the required capacity is unavailable. Each workload needs an explicit decision.
Two replicas are useful only when they fail independently
Running two replicas of an important web application is useful only if several other conditions are true:
- the replicas land on different physical failure domains;
- the surviving host has enough spare CPU and memory;
- the ingress path remains available;
- the application can safely run more than one replica;
- its database and files remain available after the failure.
Otherwise, the second replica may only create the appearance of redundancy.
The same rule applies to cluster services. Traefik or another ingress controller must have replicas placed across the two worker domains. Argo CD needs either an availability design or an explicitly accepted recovery time. A highly available application is not useful if the only ingress instance disappeared with the failed host.
Control-plane HA is a separate problem
The three k3s server nodes run on separate physical mini PCs, which is a better failure-domain layout than the workers currently have. An embedded-etcd k3s cluster uses an odd number of server nodes so etcd can maintain quorum; the official k3s High Availability documentation describes this requirement.
With three healthy server nodes, the control plane can tolerate the loss of one. Losing two means losing etcd quorum, even if the remaining server process still appears to be running.
That leads to an important operational distinction:
A healthy control plane does not guarantee healthy workloads, and running worker VMs do not guarantee a healthy control plane.
Both layers need their own failure scenarios and tests.
Storage remains another failure domain
Moving workers across two Proxmox hosts does not automatically make stateful workloads highly available.
Some application data is stored on shared NAS-backed volumes, while other data may be tied to node-local storage. Those cases fail differently:
- If a worker disappears, a shared volume may still be reachable from another worker.
- If the NAS disappears, Pods can remain Running while the application loses access to its data.
- If a local volume is tied to the failed worker, the scheduler cannot recreate that data on another node.
For every stateful service, the migration plan therefore needs to identify the database, persistent files, storage class, backup method and tested restore path. “It has a PVC” is not a disaster-recovery plan.
Capacity planning for a host failure
Splitting workers evenly is not enough if both Proxmox hosts normally run near their limits.
If host A fails, host B must have enough free capacity for the critical workloads that were running on A. The reverse must also be true. This does not necessarily mean reserving enough capacity for every development workload, but it does require a priority list.
My practical categories are:
- Must survive automatically: ingress and important internal applications.
- May restart with a delay: lower-priority development and test services.
- May remain unavailable during recovery: nonessential tools and disposable jobs.
Resource requests and limits have to reflect reality for the scheduler to make useful decisions. If requests are missing or unrealistic, a failover may schedule successfully on paper and then overload the surviving host.
Keep the bootstrap dependencies outside the cluster
One deliberate boundary in this design is keeping the main GitLab installation outside k3s for now.
GitLab contains the repositories used by Argo CD to rebuild the desired state of the cluster. Running GitLab inside the same cluster would create an uncomfortable recovery loop:
k3s fails
↓
GitLab and manifests are unavailable
↓
Argo CD cannot read the desired state
↓
the cluster needs the services that depend on the cluster
Keeping GitLab on a separate VM gives the recovery process an external anchor:
GitLab remains available
↓
rebuild or restore k3s
↓
install/connect Argo CD
↓
synchronize manifests
↓
restore application data
Not every service belongs in Kubernetes simply because it can run there. Sometimes the more resilient architecture is the one that preserves a useful boundary.
The implementation order
The planned sequence is intentionally conservative:
- Remove or archive confirmed obsolete VMs to free capacity.
- Create the additional worker VMs on the second Proxmox host.
- Join them to k3s and apply consistent failure-domain labels.
- Verify node capacity, storage access and network paths.
- Add topology rules to selected critical workloads.
- Spread ingress replicas across both physical hosts.
- Test the loss of one worker and then a controlled loss of one Proxmox host.
- Only after those tests, continue migrating more applications.
The original VMs for migrated applications should not be deleted until backup and restore of the new platform have been verified.
What “HA enough” means for this cluster
This is not an attempt to reproduce a hyperscale cloud platform in a small environment. The goal is a clear and testable level of availability:
- one worker VM can fail without taking down an important replicated application;
- one Proxmox host can fail while critical stateless workloads continue on the other;
- one control-plane node can fail while etcd retains quorum;
- storage loss is treated as a separate incident, not hidden behind Kubernetes replicas;
- the GitOps source remains available outside the cluster;
- recovery assumptions are validated with controlled tests.
The main lesson
The worker count was never the right availability metric. The useful question was:
How many independent things must fail before the service becomes unavailable?
Mapping virtual nodes back to physical hosts exposed the real weakness immediately. Before moving more production services to k3s, I need to make the scheduler aware of the infrastructure beneath it and verify that the remaining host can actually carry the important workloads.
The next field note in this series will cover the other half of the migration decision: which existing services are good k3s candidates, and which ones should deliberately remain on virtual machines.