Skip to content

Secret Management#

In this repository I use two ways to encrypt secrets, both utilizing sops and age.

age is the recommended encryption tool for sops as it is more secure and easier to use than gpg.

Requirements#

Preparation#

For both variants we need two age keypairs. One for local use and one for ArgoCD.

1
2
3
4
5
6
# the folder for the age keypairs, consumed by sops
mkdir -p "$HOME/.config/sops/age"
# local age keypair
age-keygen -o "$HOME/.config/sops/age/keys.txt"
# argocd age keypair
age-keygen -o "$HOME/.config/sops/age/argo-cd.txt"

Example output:

1
2
3
4
cat "$HOME/.config/sops/age/keys.txt"
# created: 2024-05-28T07:23:28+02:00
# public key: age***
AGE-SECRET-KEY-19***

ArgoCD needs the private key of the local keypair to decrypt the secrets. So we create a kubernetes secret with the private key that ArgoCD gets mounted.

1
2
cat "$HOME/.config/sops/age/argo-cd.txt" | kubectl create secret generic sops-age --namespace=argocd \
--from-file=keys.txt=/dev/stdin

Adjusted helm values to mount the sops-age secret into the argocd-server pod:

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
repoServer:
  volumes:
    - name: sops-age
      secret:
        secretName: sops-age
  volumeMounts:
    - mountPath: /.config/sops/age
      name: sops-age
      readOnly: true
  env:
    - name: SOPS_AGE_KEY_FILE
      value: /.config/sops/age/keys.txt

Repository configuration#

Info

The secrets need to be encrypted with both public keys. The ArgoCD key is used to decrypt the secrets in the ArgoCD cluster and the local key is used to de- and encrypt the secrets locally.

Create a .sops.yaml file in the repository root. Example:

.sops.yaml
1
2
3
4
5
6
creation_rules:
  - path_regex: .*.enc.yaml
    encrypted_regex: "^(data|stringData|email|dnsNames|.*(H|h)osts?|hostname|username|password|url|issuer|clientSecret|argocdServerAdminPassword|oidc.config|commonName|literals)$"
    age: age1d2g7tgqpfvxulsusn3m608h60h2hne7yqwv5nh5nd24z6h0hgq0skjkhw8,age1q522xtgjrmvr43w7um5rh02ta3yfns635680hz4m7uhw0nfqj5zqgxnz27
  - path_regex: secrets/argo-cd.age
    age: age1d2g7tgqpfvxulsusn3m608h60h2hne7yqwv5nh5nd24z6h0hgq0skjkhw8

The .sops file for this repository differs from this example.

Kubernetes Secrets / Manifests via KSOPS#

To use sops with ArgoCD, you need to mount ksops and the sops-age key into the argocd-server pod. The following helm values start the ksops container as an initContainer, copies the ksops and kustomize binaries into the custom-tools volume and mounts the binaries from the custom-tools volume into the repo server container. The sops-age key is also mounted into the repo server container.

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
configs:
  cm:
    kustomize.buildOptions: "--enable-helm --enable-alpha-plugins --enable-exec"
repoServer:
  # Use init containers to configure custom tooling
  # https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
  volumes:
    - name: custom-tools
      emptyDir: {}
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.3.1
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    # ksops packages it's own kustomize binary with ksops integration, overrides the argocd kustomize binary
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
  env:
    - name: XDG_CONFIG_HOME
      value: /.config

Adjusting kustomize configuration#

Info

To tell kustomize to use ksops for decryption, we need to add a generators configuration to the kustomization.yaml file.

kustomization.yaml
1
2
generators:
  - kustomize-secret-generator.yaml
kustomize-secret-generator.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - backup-secrets.enc.yaml

The backup-secrets.enc.yaml is just a normal kubernetes manifest but with sops encrypted values: Example emby backup-secrets.enc.yaml. Do not add thouse manifests to the resource list of the kustomization.yaml because this would result in a duplicate resource error.

Kustomize managed Helm values#

This repository makes heavy use of kustomize rendering helm charts. Kustomize can manage helm values either directly in the kustomization.yaml or in a separate file. The helm values can contain sensitive values and it's not possible to encrypt values in the kustomization.yaml file directly so we need to use a separate helm values file.

Info

The workflow basically is to create an encrypted values.enc.yaml, tell kustomize to get the helm values from values.yaml and use an ArgoCD ConfigManagementPlugin to decrypt the values.enc.yaml to values.yaml. The ConfigManagementPlugin gets executed when ArgoCD finds a values.enc.yaml before kustomize renders the kubernetes manifests.

Configuration of the ConfigManagementPlugin#

If ArgoCD finds a values.enc.yaml in an application directory, argo-cd runs the CMP cmp-sops-decrypt, which decryptes the file to values.yaml, and then runs kustomize.

The ConfigManagementPlugin is configured as a configmap within a separate argo-cd app deployment. Additionaly it needs helm and sops binaries for the argo-cd repo-server which get configured via a sidecar container. So the deployment of it needs to be extended.

ConfigManagementPlugin:

argocd-cmp-sops-plugin.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmp-sops-plugin
  namespace: argocd
data:
  plugin.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: cmp-sops-decrypt
    spec:
      version: v1.0
      generate:
        command: [sh, -c]
        args:
          - sops --decrypt --input-type yaml --output-type yaml values.enc.yaml > values.yaml;
            kustomize build --enable-helm --enable-alpha-plugins --enable-exec .
      discover:
        fileName: "values.enc.yaml"

The adjusted helm values for the argo-cd repo-server. Some of the adjustments are dependent on changes from the previous section (like ksops and kustomize usage).

values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
repoServer:
  # Addingcustom tools volume and ConfigManagementPlugin to the repo-server pod deployment
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: cmp-tmp
      emptyDir: {}
    - name: cmp-sops-plugin
      configMap:
        name: argocd-cmp-sops-plugin
  # Installation of binaries
  initContainers:
    - name: install-sops
      image: ghcr.io/getsops/sops:v3.8.1-alpine
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing SOPS...";
          cp /usr/local/bin/sops /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: install-helm
      image: alpine/helm:3.15.1
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing helm..."; cp /usr/bin/helm /custom-tools/; echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  # Adding Container responsible for the configured ConfigManagementPlugin
  extraContainers:
    - name: cmp-sops-plugin
      command:
        - "/var/run/argocd/argocd-cmp-server"
      image: alpine:3.20.0
      imagePullPolicy: IfNotPresent
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: plugin.yaml
          name: cmp-sops-plugin
        - mountPath: /tmp
          name: cmp-tmp
        - mountPath: /usr/local/bin/kustomize
          name: custom-tools
          subPath: kustomize
        - mountPath: /usr/local/bin/ksops
          name: custom-tools
          subPath: ksops
        - mountPath: /usr/local/bin/sops
          name: custom-tools
          subPath: sops
        - mountPath: /usr/local/bin/helm
          name: custom-tools
          subPath: helm
        - mountPath: /.config/sops/age
          name: sops-age
          readOnly: true
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
      readOnly: true

This basically builds the plugin container with all required tools on-demand.

Of course, the configuration could be way shorter if a container, that already includes the following binaries, would be used as extraContainer 🤷

  • kustomize (from ksops)
  • ksops
  • sops
  • helm

More information about my journey to en- and decrypt values.yaml can be found in the following ksops issue on github: Support kustomize helmCharts valuesFile.

En- and decrypting helm values and manifests#

To encrypt a file inplace, use the following command:

1
sops -e -i secret.enc.yaml

To decrypt a file inplace, use the following command:

1
sops -d -i secret.enc.yaml

Tip

If you're working with VSCode I can recommend the extension @signageos/vscode-sops which automatically decrypts and encrypts secrets on save.

It can also automatically encrypt files which are not yet encrypted. To enable this feature, add the following to your settings.json:

1
2
3
{
  "sops.creationEnabled": true
}

This will automatically encrypt files which match the creation_rules in the .sops.yaml file.