Skip to content

Secret Management#

In this repository I use multiple ways to encrypt secrets, utilizing sops and age.

age is the recommended encryption tool for sops as it is more secure and easier to use than gpg.

Requirements#

Preparation#

For all variants we need two age keypairs. One for local use and one for ArgoCD.

1
2
3
4
5
6
# the folder for the age keypairs, consumed by sops
mkdir -p "$HOME/.config/sops/age"
# local age keypair
age-keygen -o "$HOME/.config/sops/age/keys.txt"
# argocd age keypair
age-keygen -o "$HOME/.config/sops/age/argo-cd.txt"

Example output:

1
2
3
4
cat "$HOME/.config/sops/age/keys.txt"
# created: 2024-05-28T07:23:28+02:00
# public key: age***
AGE-SECRET-KEY-19***

ArgoCD needs the private key of the local keypair to decrypt the secrets. So we create a kubernetes secret with the private key that ArgoCD gets mounted.

1
2
cat "$HOME/.config/sops/age/argo-cd.txt" | kubectl create secret generic sops-age --namespace=argocd \
--from-file=keys.txt=/dev/stdin

Adjusted helm values to mount the sops-age secret into the argocd-server pod:

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
repoServer:
  volumes:
    - name: sops-age
      secret:
        secretName: sops-age
  volumeMounts:
    - mountPath: /.config/sops/age
      name: sops-age
      readOnly: true
  env:
    - name: SOPS_AGE_KEY_FILE
      value: /.config/sops/age/keys.txt

Repository configuration#

Info

The secrets need to be encrypted with both public keys. The ArgoCD key is used to decrypt the secrets in the ArgoCD cluster and the local key is used to de- and encrypt the secrets locally.

Create a .sops.yaml file in the repository root. Example:

.sops.yaml
1
2
3
4
5
6
creation_rules:
  - path_regex: .*.enc.yaml
    encrypted_regex: "^(data|stringData|email|dnsNames|.*(H|h)osts?|hostname|username|password|url|issuer|clientSecret|argocdServerAdminPassword|oidc.config|commonName|literals)$"
    age: age1d2g7tgqpfvxulsusn3m608h60h2hne7yqwv5nh5nd24z6h0hgq0skjkhw8,age1q522xtgjrmvr43w7um5rh02ta3yfns635680hz4m7uhw0nfqj5zqgxnz27
  - path_regex: secrets/argo-cd.age
    age: age1d2g7tgqpfvxulsusn3m608h60h2hne7yqwv5nh5nd24z6h0hgq0skjkhw8

The .sops file for this repository differs from this example.

Kubernetes Secrets / Manifests via KSOPS#

To use sops with ArgoCD, you need to mount ksops and the sops-age key into the argocd-server pod. The following helm values start the ksops container as an initContainer, copies the ksops and kustomize binaries into the custom-tools volume and mounts the binaries from the custom-tools volume into the repo server container. The sops-age key is also mounted into the repo server container.

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
configs:
  cm:
    kustomize.buildOptions: "--enable-helm --enable-alpha-plugins --enable-exec"
repoServer:
  # Use init containers to configure custom tooling
  # https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.3.1
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    # ksops packages it's own kustomize binary with ksops integration, overrides the argocd kustomize binary
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
  env:
    - name: XDG_CONFIG_HOME
      value: /.config

Adjusting kustomize configuration#

Info

To tell kustomize to use ksops for decryption, we need to add a generators configuration to the kustomization.yaml file.

kustomization.yaml
1
2
generators:
  - kustomize-secret-generator.yaml
kustomize-secret-generator.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - backup-secrets.enc.yaml

The backup-secrets.enc.yaml is just a normal kubernetes manifest but with sops encrypted values: Example emby backup-secrets.enc.yaml. Do not add thouse manifests to the resource list of the kustomization.yaml because this would result in a duplicate resource error.

Kustomize managed Helm values#

This repository makes heavy use of kustomize rendering helm charts. Kustomize can manage helm values either directly in the kustomization.yaml or in a separate file. The helm values can contain sensitive values and it's not possible to encrypt values in the kustomization.yaml file directly so we need to use a separate helm values file.

There are two patterns for managing Helm values with SOPS encryption, depending on whether the values file contains only secrets or a mix of secrets and non-sensitive configuration.

Pattern 1: Legacy - All values encrypted (values.enc.yaml only)#

Info

The original workflow creates an encrypted values.enc.yaml, tells kustomize to get the helm values from values.yaml and uses an ArgoCD ConfigManagementPlugin to decrypt the values.enc.yaml to values.yaml. The ConfigManagementPlugin gets executed when ArgoCD finds a values.enc.yaml before kustomize renders the kubernetes manifests.

kustomization.yaml
1
2
3
helmCharts:
  - name: my-chart
    valuesFile: values.yaml  # Generated by CMP from values.enc.yaml

Pattern 2: Split values (values.public.yaml + values.enc.yaml)#

To allow Renovate to update image tags and other non-sensitive values, the values can be split into two files:

  • values.public.yaml - Non-sensitive values (image tags, resource limits, etc.) committed in plaintext. Renovate can read and update this file.
  • values.enc.yaml - Only SOPS-encrypted secret values (passwords, tokens, hostnames, etc.)

The CMP detects the presence of values.public.yaml and decrypts values.enc.yaml to values.secrets.yaml instead of overwriting values.yaml. Kustomize then merges both files via additionalValuesFiles (requires kustomize 5.0+):

kustomization.yaml
1
2
3
4
5
helmCharts:
  - name: my-chart
    valuesFile: values.public.yaml
    additionalValuesFiles:
      - values.secrets.yaml  # Generated by CMP from values.enc.yaml

If an app has no secrets at all in its Helm values, it only needs values.public.yaml without a values.enc.yaml. The CMP will not trigger (discover rule) and ArgoCD uses the standard kustomize builder.

Note

values.yaml is listed in .gitignore to prevent accidentally committing decrypted secrets. The naming convention values.public.yaml makes the intent explicit: this file is safe to commit.

Configuration of the ConfigManagementPlugin#

If ArgoCD finds a values.enc.yaml in an application directory, it runs the CMP cmp-sops-decrypt. The CMP supports both patterns:

  • If values.public.yaml exists: decrypts values.enc.yaml to values.secrets.yaml (split pattern)
  • If values.public.yaml does not exist: decrypts values.enc.yaml to values.yaml (legacy pattern)

The ConfigManagementPlugin is configured as a configmap within a separate argo-cd app deployment. Additionally it needs helm and sops binaries for the argo-cd repo-server which get configured via a sidecar container. So the deployment of it needs to be extended.

ConfigManagementPlugin:

argocd-cmp-sops-plugin.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmp-sops-plugin
  namespace: argocd
data:
  plugin.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: cmp-sops-decrypt
    spec:
      version: v1.1
      generate:
        command: [sh, -c]
        args:
          - |
            if [ -f values.enc.yaml ]; then
              if [ -f values.public.yaml ]; then
                sops --decrypt --input-type yaml --output-type yaml values.enc.yaml > values.secrets.yaml;
              else
                sops --decrypt --input-type yaml --output-type yaml values.enc.yaml > values.yaml;
              fi;
            fi;
            kustomize build --enable-helm --enable-alpha-plugins --enable-exec .
      discover:
        fileName: "values.enc.yaml"

The adjusted helm values for the argo-cd repo-server. Some of the adjustments are dependent on changes from the previous section (like ksops and kustomize usage).

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
repoServer:
  # Addingcustom tools volume and ConfigManagementPlugin to the repo-server pod deployment
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: cmp-tmp
      emptyDir: {}
    - name: cmp-sops-plugin
      configMap:
        name: argocd-cmp-sops-plugin
  # Installation of binaries
  initContainers:
    - name: install-sops
      image: ghcr.io/getsops/sops:v3.8.1-alpine
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing SOPS...";
          cp /usr/local/bin/sops /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: install-helm
      image: alpine/helm:3.15.1
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing helm..."; cp /usr/bin/helm /custom-tools/; echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  # Adding Container responsible for the configured ConfigManagementPlugin
  extraContainers:
    - name: cmp-sops-plugin
      command:
        - "/var/run/argocd/argocd-cmp-server"
      image: alpine:3.20.0
      imagePullPolicy: IfNotPresent
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: plugin.yaml
          name: cmp-sops-plugin
        - mountPath: /tmp
          name: cmp-tmp
        - mountPath: /usr/local/bin/kustomize
          name: custom-tools
          subPath: kustomize
        - mountPath: /usr/local/bin/ksops
          name: custom-tools
          subPath: ksops
        - mountPath: /usr/local/bin/sops
          name: custom-tools
          subPath: sops
        - mountPath: /usr/local/bin/helm
          name: custom-tools
          subPath: helm
        - mountPath: /.config/sops/age
          name: sops-age
          readOnly: true
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
      readOnly: true

This basically builds the plugin container with all required tools on-demand.

Of course, the configuration could be way shorter if a container, that already includes the following binaries, would be used as extraContainer 🤷

  • kustomize (from ksops)
  • ksops
  • sops
  • helm

More information about my journey to en- and decrypt values.yaml can be found in the following ksops issue on github: Support kustomize helmCharts valuesFile.

Multi-source Helm values with helm-secrets#

The CMP-based pattern only works for single-source ArgoCD Applications where ArgoCD invokes Kustomize on a directory inside this repository. ArgoCD also supports multi-source Applications where one source is an external Helm chart repository (or a Git repository containing the chart) and a second source is this repository providing values via a $values reference. In that flow ArgoCD calls helm template directly — no Kustomize, no CMP — so values.enc.yaml would be passed to Helm without decryption.

For this case the helm-secrets Helm plugin is integrated into the argocd-repo-server. It is wrapped around the helm binary so any helm install, helm upgrade, helm template, helm lint or helm diff invocation is intercepted and routed through helm secrets, which transparently decrypts SOPS-encrypted value files before passing them to Helm.

Why a wrapper is needed#

HELM_SECRETS_WRAPPER_ENABLED=true alone is not enough. The wrapper is a small shell script (scripts/wrapper/helm.sh) that ships with helm-secrets and rewrites Helm subcommands. It must be placed in PATH before the real Helm binary so it is the first match for helm. The wrapper itself then calls the real Helm via $HELM_BIN to avoid recursion.

The argocd-repo-server has the standard PATH:

1
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

The real Helm binary lives at /usr/local/bin/helm. Mounting the wrapper as /usr/local/sbin/helm lets it shadow the real binary without touching the original.

Repo-server configuration#

In addition to the existing custom-tools volume and SOPS / age setup, add an init container that downloads helm-secrets and stages the wrapper as a standalone file. Then mount the plugin directory (so helm plugin list resolves secrets) and the wrapper file:

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
repoServer:
  initContainers:
    - name: install-helm-secrets
      # https://github.com/jkroepke/helm-secrets/releases
      image: alpine:3.21
      command:
        - /bin/sh
        - -c
      args:
        - mkdir -p /custom-tools/helm-plugins && wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v4.7.6/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf- && cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm-wrapper && chmod +x /custom-tools/helm-wrapper
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /helm-plugins
      name: custom-tools
      subPath: helm-plugins
    # The wrapper is mounted in /usr/local/sbin so it precedes /usr/local/bin
    # in PATH and is found first when something invokes `helm`.
    - mountPath: /usr/local/sbin/helm
      name: custom-tools
      subPath: helm-wrapper
  env:
    - name: HELM_PLUGINS
      value: /helm-plugins/
    - name: HELM_SECRETS_BACKEND
      value: sops
    - name: HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH
      value: "true"
    - name: HELM_SECRETS_WRAPPER_ENABLED
      value: "true"
    # The wrapper invokes the real helm via $HELM_BIN to avoid recursion.
    - name: HELM_BIN
      value: /usr/local/bin/helm

HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH=true is required so helm-secrets accepts the absolute paths that ArgoCD generates when it materializes $values/... references into the repo-server filesystem.

Application manifest pattern#

A multi-source Application points the chart source at the upstream Helm/Git repo and references the encrypted values file via $values:

apps/argo-cd-apps/13-pihole03.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: 13-pihole03
  namespace: argocd
spec:
  sources:
    - repoURL: https://github.com/madic-creates/pihole-kubernetes.git
      targetRevision: HEAD
      path: charts/pihole
      helm:
        releaseName: pihole-03
        valueFiles:
          - $values/apps/pihole-03/helm-values.enc.yaml
    - repoURL: "example.com"  # replaced via kustomize from a sealed secret
      targetRevision: HEAD
      ref: values

No secrets:// URL prefix is used. With HELM_SECRETS_WRAPPER_ENABLED=true the wrapper detects the SOPS-encrypted file from its content and decrypts it on the fly. The prefix-style references documented by helm-secrets do not work with multi-source apps because ArgoCD requires the $values/ source reference to be at the start of the path string.

En- and decrypting helm values and manifests#

To encrypt a file inplace, use the following command:

1
sops -e -i secret.enc.yaml

To decrypt a file inplace, use the following command:

1
sops -d -i secret.enc.yaml

Tip

If you're working with VSCode I can recommend the extension @signageos/vscode-sops which automatically decrypts and encrypts secrets on save.

It can also automatically encrypt files which are not yet encrypted. To enable this feature, add the following to your settings.json:

1
2
3
{
  "sops.creationEnabled": true
}

This will automatically encrypt files which match the creation_rules in the .sops.yaml file.