Skip to content

Update management#

The cluster uses two components to manage updates:

Renovate#

Renovate is configured as a scheduled github workflow. Additionally, the action can be run manually with optional increased debug log or in dry-run mode. As RENOVATE_TOKEN a github personal access token is used.

More information about the required token permissions can be found in the official docs: https://docs.renovatebot.com/modules/platform/github/. I keep it here because I had a hard time finding thoses information 🙈

Rules#

Multiple configurations in the renovate.json support renovate in finding updates.

Kubernetes Manifests

Renovate identifies Kubernetes manifests depending on the file name. The current regex is apps\/.+\/k8s\\..*\\.yaml$. Basically beneath the apps folder it matches every yaml file that begins with k8s.

Danger

Renovate must not change Kubernetes manifests which contain sops encrypted entries because that would break the file signature. Encrypted Kubernetes manifests aren't allowed to begin with k8s!

Auto Updates

Renovate will auto-update every entry after 5 days of release. If not explicitly excluded, every minor and patch release gets automatically updated / merged. Major releases need to be approved via Pull Request.

Longhorn is an exception. It will never automerge and will only create Pull Requests for releases that are available for 14 days, even for patch releases. Because Longhorn is the storage provider for the cluster, I want it to be a tested release.

Image Tags

Beside updating image tags in kubernetes manifests, renovate updates image tags in kustomization.yaml files when it finds the keyword image or the following structure:

1
2
repository: emby/embyserver
tag: 4.9.0.30

Repository is required for renovate to know, in which repository to search for new versions to update the tag.

Image Tags in Helm Values (values.public.yaml)

Renovate cannot read SOPS-encrypted files (values.enc.yaml). To allow Renovate to update image tags in Helm values, non-sensitive values are extracted into a plaintext values.public.yaml file. See Secret Management - Pattern 2 for details on the split pattern.

A custom manager in renovate.json5 scans values.public.yaml files for the same repository + tag structure:

values.public.yaml
1
2
3
image:
    repository: ghcr.io/pi-hole/pihole
    tag: 2025.11.0

pre-commit hooks

Renovate is able to update pre-commit hooks. But it's still in beta and can lead to problems. Because of this automerge for pre-commit is disabled and every update requires approval through a pull request.

Dependency Dashboard#

Renovate keeps its state in the "Dependency Dashboard" issue. The dashboard can lag behind main by several hours, for example still listing an old chart version under "Detected Dependencies" or showing a PR as open after it has been merged. That is how Renovate works, not a sign that the workflow is broken.

Renovate only rewrites the dashboard when a repository run finishes without an automerge. When a branch automerges, Renovate restarts the run once on the new main. If the restarted run automerges again, it stops without touching the dashboard ("Automerged but already retried once", see lib/workers/repository/index.ts in Renovate 44). With many minor/patch automerges queued, several scheduled runs in a row can end like this, and the dashboard stays at the last run that had nothing left to automerge.

To check whether the dashboard is just behind:

  • Compare the issue's last update time with the latest workflow runs. Each run logs "result": "automerged" or "result": "done" under "Repository finished". Only done runs update the dashboard.
  • The dashboard catches up on its own once a run has nothing left to automerge. To force that sooner, trigger the workflow manually (repeat if that run also automerges).

System Upgrade Controller#

The System Upgrade Controller is a Kubernetes controller that manages the upgrade of a Kubernetes cluster. There are only plans configured to automatically upgrade the cluster to the latest k3s version.

The update plans are a separate ArgoCD app because they require CRDs from the system-upgrade-controller. Without those CRDs ArgoCD would not be able to apply the plans and the deployment would fail if they are included in the system-upgrade-controller app.