CCE Secrets Manager for DEW

Introduction

The CCE Secrets Manager for DEW add-on (formerly dew-provider) is used to interconnect with Data Encryption Workshop (DEW), which allows you to mount secrets stored outside a cluster (DEW for storing sensitive information) to pods. In this way, sensitive information can be decoupled from the cluster environment, which prevents information leakage caused by program hardcoding or plaintext configuration. With this add-on, you can enjoy the following capabilities:

Mounting secrets: You can mount secrets in DEW to service pods, reducing information leakage risks.
Scheduled rotation: After a pod runs normally, if the secret declared in the SecretProviderClass object and stored in CSMS is updated, the latest secret values can be updated to the pod through scheduled rotation. When using this capability, set the secret version to latest.
Real-time detection of SecretProviderClass changes: After a pod runs normally, if a user modifies the secret declared in the SecretProviderClass object (for example, a secret is added or the version number is changed), the add-on can detect the change in real time and update the secret to the pod.

Notes and Constraints

DEW includes Key Management Service (KMS), Cloud Secret Management Service (CSMS), and Key Pair Service (KPS). Currently, the dew-provider add-on can interconnect only with CSMS.
A maximum of 500 SecretProviderClass objects can be created.
When the add-on is uninstalled, related CRD resources are deleted accordingly. Even if the add-on is reinstalled, the original SecretProviderClass object is unavailable. If you want to use the original SecretProviderClass resources after the add-on is uninstalled and then reinstalled, manually create them again.

Installing the Add-on

Log in to the CCE console and click the cluster name to access the cluster console. In the navigation pane, choose Add-ons, locate CCE Secrets Manager for DEW on the right, and click Install.

On the Install Add-on page, configure parameters in the Parameters area, as listed in the following table.

Item	Parameter	Description
Secret Synchronization Period	rotation_poll_interval	Rotation interval, in unit of m (instead of min). The rotation interval indicates the interval for sending a request to CSMS and obtaining the latest secret. The proper interval range is [1m, 1440m]. The default value is 2m.

Click Install.
After the add-on is installed, select the cluster and choose Add-ons in the navigation pane. On the displayed page, view the add-on in the Add-ons Installed area.

Components

Table 1 Add-on components
Component	Description	Resource Type
dew-provider	A component that obtains specified secrets from CSMS and mounts them to the pods	DaemonSet
csi-secrets-store	A component that maintains two CRDs, SecretProviderClass and SecretProviderClassPodStatus (spcPodStatus). SecretProviderClass is used to describe the required secret (such as the secret version and name). It is created by users and will be referenced in pods. spcPodStatus is used to trace the binding relationships between pods and secrets. It is automatically created by csi-driver and requires no manual operation. One pod corresponds to one spcPodStatus. After a pod is started, a spcPodStatus is generated for the pod. When the pod lifecycle ends, the spcPodStatus is deleted accordingly.	DaemonSet

Mounting Secrets

To mount a secret in DEW to a container, you need to take the following steps: First, create a ServiceAccount object as the secret for the cluster to access DEW. Then, create a SecretProviderClass object to define and obtain the secret in DEW. Finally, mount the ServiceAccount and SecretProviderClass objects to the service pod to ensure that the pod can securely access the secret. Currently, there are two methods to mount a secret to a pod:

Mounting a secret through a volume: Secrets are mounted to containers in the form of volumes. This method is suitable when there is a large amount of data.
Mounting a secret created from a secret in DEW: Secrets are mounted to containers as environment variables. This method is suitable when there is only a small amount of sensitive data.

The following describes the processes of the two methods.

Mounting a Credential Using a Volume

Create a ServiceAccount object to declare the name of the secret that can be used by the service pod. Before you take this step, ensure that the corresponding secret exists in CSMS. If the secret does not exist, an error occurs when the secret is obtained from CSMS. As a result, the pod cannot run normally.
1. Create a YAML file for the ServiceAccount object.
```
vim serviceaccount.yaml
```
  The following is the file content. You can declare the secret names that can be used by the service pod in the cce.io/dew-resource field. In this example, secret_1 and secret_2 can be referenced in the service pod. If you reference a secret that is not declared here, the mounting will fail. As a result, the pod cannot run normally.
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-spc-sa
  annotations:
    cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use
```
2. Create the ServiceAccount object.
```
kubectl apply -f serviceaccount.yaml
```
  Information similar to the following is displayed:
```
serviceaccount/nginx-spc-sa created
```
3. Check whether the ServiceAccount object has been created.
```
kubectl get sa
```
  If the following information is displayed, the ServiceAccount object named nginx-spc-sa has been created and will be referenced in the service pods.
```
NAME            SECRETS   AGE
default         1         18d   # This is the default ServiceAccount object of the system.
nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.
```

Create a SecretProviderClass object to define and manage secrets from DEW so that secrets can be securely mounted to pods in the CCE cluster.

Create a YAML file for the SecretProviderClass object.

vim serviceaccount.yaml

The following is the file content. The parameters.objects field specifies the secrets to be mounted, in array format.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce     # The value is fixed at cce.
  parameters:
    objects: |
          - objectName: "secret_1"
            objectAlias: "secret_test"
            objectVersion: "v1"
            objectType: "csms"

Table 2 objects parameters
Parameter	Type	Mandatory	Description
objectName	String	Yes	Secret name. Set this parameter to the secrets referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectAlias	String	No	File name of the secret written into the container. If this parameter is not specified, the name of the file that the secret is written to the container is objectName by default. If this parameter is specified, the value of objectAlias must be different from that of objectName and of objectAlias of other secrets and its own objectName. Otherwise, the mounting fails.
objectType	String	Yes	Type of the secret. Only csms is supported. Any value other than csms is invalid.
objectVersion	String	Yes	Version of the secret. Specify a version, for example, v1 or v2. Use the latest version, for example, latest. If objectVersion is set to latest and the corresponding secret in CSMS is updated, scheduled rotation is automatically enabled, and the secret will be updated to the pod after the interval specified by rotation_poll_interval.

Create the SecretProviderClass object.
```
kubectl apply -f secretproviderclass.yaml
```
Information similar to the following is displayed:
```
secretproviderclass/spc-test created
```
Check whether the SecretProviderClass object has been created.
```
kubectl get spc
```
If the following information is displayed, a SecretProviderClass object named spc-test has been created and will be referenced in subsequent pods.
```
NAME   AGE
spc-test   20h
```

Create a service pod. In this example, this pod is used to run an Nginx application.

Define a workload, reference the created ServiceAccount object in serviceAccountName, and reference the created SecretProviderClass object in secretProviderClass, specify the mount path of the container in mountPath. (Do not specify special directories such as / and /var/run. Otherwise, the container may fail to be started.)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-spc
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      serviceAccountName: nginx-spc-sa   # Reference the created ServiceAccount for identity authentication of secrets.
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io   # Specify the Secrets Store CSI driver. The value is fixed.
            readOnly: true                     # Set the volume to read-only to ensure that the secret is not modified.
            volumeAttributes:
              secretProviderClass: "spc-test"  # Reference the created SPC.
      containers:
        - name: nginx-spc
          image: nginx:alpine
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"  # Define the mount path of secrets in the container.
              readOnly: true
      imagePullSecrets:
        - name: default-secret

Create a pod.

kubectl apply -f deployment.yaml

Information similar to the following is displayed:

deployment/nginx-spc created

Check whether the pod has been created.

kubectl get pod

If STATUS of the pod is Running in the command output, the pod has been created.

NAME                           READY   STATUS     RESTARTS     AGE
nginx-spc-67c9d5b594-642np     1/1     Running    0            20s

Check whether the specified secret can be written.

Enter the container.

kubectl exec -ti nginx-spc-67c9d5b594-642np -- /bin/sh

Search for the specified secret.
```
cd /mnt/secrets-store/
ls
```
If information similar to the following is displayed, secret_1 declared in the SecretProviderClass object has been written to the pod.
```
secret_1
```

Obtain spcPodStatus to check whether the secret has been mounted to the pod.

Exit the container using Ctrl+d and obtain the spcps resource name.

kubectl get spcps

Information similar to the following is displayed:

NAME                                          AGE
nginx-spc-67c9d5b594-642np-default-spc-test   103s

Check whether the secret has been mounted to the pod.

kubectl get spcps nginx-spc-67c9d5b594-642np-default-spc-test -o yaml

If information similar to the following is displayed, the secret has been mounted to the pod.

......
status:
mounted: true
objects:    # Mounted secret
- id: secret_1
version: v1
podName: nginx-spc-67c9d5b594-642np   # Pod that references the SecretProviderClass object
secretProviderClassName: spc-test               # SPC object
targetPath: /mnt/paas/kubernetes/kubelet/pods/6dd29596-5b78-44fb-9d4c-a5027c420617/volumes/kubernetes.io~csi/secrets-store-inline/mount

Mounting a Credential Using a Secret

Note

CCE Secrets Manager for DEW must be 1.1.1 or later.

Create a ServiceAccount object to declare the name of the secret that can be used by the service pod. Before you take this step, ensure that the corresponding secret exists in CSMS. If the secret does not exist, an error occurs when the secret is obtained from CSMS. As a result, the pod cannot run normally.
1. Create a YAML file for the ServiceAccount object.
```
vim serviceaccount.yaml
```
  The following is the file content. You can declare the secret names that can be used by the service pod in the cce.io/dew-resource field. In this example, secret_1 and secret_2 can be referenced in the service pod. If you reference a secret that is not declared here, the mounting will fail. As a result, the pod cannot run normally.
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-spc-sa
  annotations:
    cce.io/dew-resource: "[\"secret_1\",\"secret_2\"]"  #secrets that allow pod to use
```
2. Create the ServiceAccount object.
```
kubectl apply -f serviceaccount.yaml
```
  Information similar to the following is displayed:
```
serviceaccount/nginx-spc-sa created
```
3. Check whether the ServiceAccount object has been created.
```
kubectl get sa
```
  If the following information is displayed, the ServiceAccount object named nginx-spc-sa has been created and will be referenced in the service pods.
```
NAME            SECRETS   AGE
default         1         18d   # This is the default ServiceAccount object of the system.
nginx-spc-sa    1         19s   # This is the newly created ServiceAccount object.
```

Create a SecretProviderClass.

Create the secretproviderclass.yaml file using the template below.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: nginx-deployment-spc-k8s-secrets
spec:
  provider: cce
  parameters: 
    # Reference a secret in CSMS.
    objects: |
      - objectName: "secret_1"
        objectType: "csms"
        objectVersion: "latest"
        jmesPath:
          - path: key1
            objectAlias: dbusername01
          - path: key2
            objectAlias: dbusername02
  # Create a CCE secret based on the CSMS secret and mount the secret to a pod.
  secretObjects:                
    - secretName: my-secret-01
      type: Opaque
      data:
        - objectName: dbusername01
          key: db_username_01
        - objectName: dbusername02
          key: db_username_02

Table 3 objects parameter
Parameter	Type	Mandatory	Description
objectName	String	Yes	Credential name. Set this parameter to the secret referenced in ServiceAccount. If there are multiple object names defined in the same SecretProviderClass, each value of the objectName parameter must be unique. Otherwise, the mounting fails.
objectType	String	Yes	Secret type. Only csms is supported. A value other than csms is invalid.
objectVersion	String	Yes	Secret version Specify a version, for example, v1 or v2. Use the latest version, for example, latest. If objectVersion is set to latest and the corresponding secret in CSMS is updated, scheduled rotation is automatically enabled, and the secret will be updated to the pod after the interval specified by rotation_poll_interval.
jmesPath	Array of Object	Yes	jmesPath is used to extract key-value pairs from objects in JSON format. CCE Secrets Manager for DEW uses this tool to support Secret mounting. If a secret has multiple keys, you need to enter multiple paths and object aliases. path: Enter the key in a DEW secret. NOTE: The key cannot contain special characters such as +, -, {}, and ()for the add-ons of versions earlier than 1.1.57. In add-ons of version 1.1.57 and later versions, the key cannot contain backslashes (\). There is no restriction on other special characters. objectAlias: name of the file mounted to a pod. The value must be the same as the value of objectName defined in secretObjects.

Table 4 secretObjects parameter
Parameter	Type	Mandatory	Description
secretName	String	Yes	Secret name
type	String	Yes	Secret type
data	Array of Object	Yes	objectName: name of the file mounted to the pod. The value must be the same as the value of objectAlias specified in objects. key: key in a secret. The value can be used to reference the encrypted content in a pod.

Run the following command to create a SecretProviderClass object:
```
kubectl apply -f secretproviderclass.yaml
```
Information similar to the following is displayed:
```
secretproviderclass/nginx-deployment-spc-k8s-secrets created
```
Check whether the SecretProviderClass object has been created.
```
kubectl get spc
```
Information similar to the following is displayed:
```
NAME                               AGE
nginx-deployment-spc-k8s-secrets   20h
```

Create a service pod. In this example, this pod is used to run an Nginx application.

vim deployment.yaml

The file content is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment-k8s-secrets
  labels:
    app: nginx-k8s-secrets
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-k8s-secrets
  template:
    metadata:
      labels:
        app: nginx-k8s-secrets
    spec:
      serviceAccountName: nginx-spc-sa   # Reference the created ServiceAccount.
      containers:
      - name: nginx-deployment-k8s-secrets
        image: nginx
        volumeMounts:   # Mount the SecretProviderClass to a container.
          - name: secrets-store-inline   # Name of the volume to be mounted
            mountPath: "/mnt/secrets"    # Path of the container to be mounted
            readOnly: true
        env:    # Set environment variables so that the container can reference the secret during running.
          - name: DB_USERNAME_01      # Environment variable name in the workload
            valueFrom:
              secretKeyRef:
                name: my-secret-01    # Secret name specified in the SecretProviderClass
                key: db_username_01   # The value of the key specified in the SecretProviderClass
          - name: DB_USERNAME_02
            valueFrom:
              secretKeyRef:
                name: my-secret-01
                key: db_username_02
      imagePullSecrets:
        - name: default-secret
      volumes:    # Use the secret specified in the SPC to create a volume.
        - name: secrets-store-inline  # Custom volume name
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: nginx-deployment-spc-k8s-secrets   # Name of the SecretProviderClass created in the previous step

Create a service pod.

kubectl apply -f deployment.yaml

Information similar to the following is displayed:

deployment/nginx-deployment-k8s-secrets created

Check whether the pod has been created.

kubectl get pod

Information similar to the following is displayed:

NAME                                              READY   STATUS     RESTARTS     AGE
nginx-deployment-k8s-secrets-6c945565fb-s7smv     1/1     Running    0            20s

Verify the result.

kubectl get secrets

The output shows that a secret named my-secret-01 has been created using secret_1 specified in the SecretProviderClass object.

NAME             TYPE                             DATA   AGE
default-secret   kubernetes.io/dockerconfigjson   1      33d
my-secret-01     Opaque                           2      1h

Scheduled Rotation

As described before, you can use this add-on to complete the mount secrets, that is, you can write the secrets stored in CSMS to a pod.

To change the secret version declared in the SPC object to latest, run the following command:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"  # change "v1"to "latest"
            objectType: "csms"

After the SPC object is updated, the add-on periodically sends a request to CSMS to obtain the value of secret_1 of the latest version and updates the value to the pod that references the SPC object. The interval for the add-on to periodically send requests is specified by rotation_poll_interval set in Installing the Add-on.

Real-Time Detection of SPC Changes

SPC changes are already detected in real time in Mounting a Credential Using a Volume and Scheduled Rotation. For demonstration, add secret secret_2 to the SPC object as follows:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: spc-test
spec:
  provider: cce
  parameters:
    objects: |
          - objectName: "secret_1"
            objectVersion: "latest"
            objectType: "csms"
          - objectName: "secret_2"
            objectVersion: "v1"
            objectType: "csms"

After the SPC object is updated, the new secret_2 is quickly mounted to the pod that references the SPC object.

Viewing Component Logs

You can use kubectl to view component logs.

View the pods for running the add-on.

kubectl get pod -n kube-system

Information similar to the following is displayed:

NAME                          READY   STATUS      RESTARTS   AGE
csi-secrets-store-76tj2       3/3     Running     0           11h
dew-provider-hm5fq            1/1     Running     0           11h

View the logs of the dew-provider component.

kubectl logs dew-provider-hm5fq -n kube-system

The command output is the detailed log information.

...
I0321 13:49:29.319708       1 dew_provider.go:43] object [secret_1]'s version [v1] doesn't change, read from local
I0321 13:49:29.319735       1 server.go:108] Start writing into mount point
I0321 13:49:29.319793       1 server.go:117] Write [secret_1/]'s value into file successfully
I0321 13:49:29.319801       1 server.go:123] Write secret file finished
I0321 13:49:29.320284       1 server.go:67] "Servicing mount request" Pod="nginx-deployment-k8s-secrets-6c945565fb-s7smv" Namespace="default" ServiceAccount="nginx-spc-sa"
I0321 13:49:29.617666       1 dew_provider.go:107] "Get secret successfully" name="secret_1" version="v1"
...

View the logs of the csi-secrets-store component. The pod of the csi-secrets-store component contains multiple containers. When viewing logs, run the -c command to specify a container. The secrets-store container is the major service container of the add-on and contains the majority of the logs.

kubectl logs csi-secrets-store-76tj2 -c secrets-store -n kube-system

The command output is the detailed log information.

...
I0321 10:17:29.314888       1 secrets-store.go:46] "Initializing Secrets Store CSI Driver" driver="secrets-store.csi.k8s.io" version="local-dev" buildTime=""
I0321 10:17:29.314987       1 reconciler.go:143] "starting rotation reconciler" rotationPollInterval="2m0s"
I0321 10:17:29.315141       1 server.go:126] "Listening for connections" address="//mnt/paas/kubernetes/kubelet/plugins/csi.sock"
I0321 10:17:30.933328       1 nodeserver.go:359] "node: getting default node info\n"
I0321 10:19:19.269337       1 nodeserver.go:353] "Using gRPC client" provider="cce" pod="nginx-spc-744b4c57dc-x74xq"
I0321 10:19:19.543221       1 secretproviderclasspodstatus_controller.go:224] "reconcile started" spcps="default/nginx-spc-744b4c57dc-x74xq-default-spc-test"
...

Parent topic: Container Security Add-ons

Предыдущая статья

Container Security Add-ons

Следующая статья

Other Add-ons

Была ли эта статья полезна?

Поддержка Юридические документы