Seamless Integration of AWS Secrets Manager with Kubernetes using Secrets Store CSI Driver (Inject Secrets as Environment Variables for EKS Workloads)

When working with sensitive credentials in Kubernetes, securely injecting AWS Secrets Manager secrets into your workloads is a priority. With AWS Secrets Store CSI Driver, you can mount secrets directly as volumes or sync them as Kubernetes secrets. This guide will walk you through integrating AWS Secrets Manager into Kubernetes, setting up IAM roles, and securely injecting secrets.

Reference: https://docs.aws.amazon.com/prescriptive-guidance/latest/secure-sensitive-data-secrets-manager-terraform/amazon-eks-secrets.html

Prerequisites

Before diving in, ensure the following:

• AWS CLI installed and configured.
• A running Kubernetes cluster.
• Required IAM permissions:CreateRole, AttachRolePolicy, CreatePolicy, CreateSecret
• OIDC Provider configured for your Kubernetes cluster (EKS recommended).
• Helm installed.

We will use the Secrets Store CSI Driver to fetch secrets from AWS Secrets Manager into Kubernetes workloads.

Steps to Integrate AWS Secrets Manager with Kubernetes

Step 1: Create an AWS Secret

First, create a secret in AWS Secrets Manager using the AWS CLI.

aws secretsmanager create-secret \
  --name demo-secrets \
  --secret-string '{"SECRET_1": "value1", "SECRET_2": "value2"}'

This creates a secret named demo-secrets with two key-value pairs: SECRET_1 and SECRET_2.

Step 2: Create an IAM Policy for Secret Access

To allow workloads to access the AWS Secrets Manager secret, create an IAM policy.

aws iam create-policy \
  --policy-name SecretsManagerReadPolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "secretsmanager:GetSecretValue"
        ],
        "Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:demo-secrets-*"
      }
    ]
  }'

Replace <region> and <account-id> with your AWS region and account ID. This policy allows read-only access to the demo-secrets secret.

Step 3: Create an IAM Role with the Policy Attached

Next, create an IAM role that Kubernetes workloads can assume to access the secret.

aws iam create-role \
  --role-name SecretsAccessRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account-id>:oidc-provider/<oidc-provider-url>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<oidc-provider-url>:sub": "system:serviceaccount:demo:demo-secrets-manager-service-account"
        }
      }
    }
  }'

• Replace <oidc-provider-url> with your cluster's OIDC provider URL.
• Replace <account-id> with your AWS account ID.
• The service account named demo-secrets-manager-service-account will be created in Step 5.

Attach the previously created policy:

aws iam attach-role-policy \
  --role-name SecretsAccessRole \
  --policy-arn arn:aws:iam::<account-id>:policy/SecretsManagerReadPolicy

Step 4: Install the Secrets Store CSI Driver and AWS Provider

Install the Secrets Store CSI Driver using Helm and enable Kubernetes secret syncing.

helm install secrets-store-csi-driver secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true

Install the AWS provider for the CSI Driver:

kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Why syncSecret.enabled=true? Setting syncSecret.enabled=true ensures the secrets are synchronized as Kubernetes secrets in addition to being available as mounted volumes.

Step 5: Create a Kubernetes Service Account

Create a Kubernetes service account with annotations pointing to the IAM role created earlier.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-secrets-manager-service-account
  namespace: demo
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/SecretsAccessRole

Apply the manifest:

kubectl apply -f service-account.yaml

Step 6: Create the SecretProviderClass

Define a SecretProviderClass that specifies which secrets to fetch from AWS Secrets Manager.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: demo-secrets
  namespace: demo
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "demo-secrets"
        objectType: "secretsmanager"
        jmesPath:
          - path: "SECRET_1"
            objectAlias: "SECRET_1"
          - path: "SECRET_2"
            objectAlias: "SECRET_2"
  secretObjects:
    - secretName: demo-secrets
      type: Opaque
      data:
        - objectName: "SECRET_1"
          key: "SECRET_1"
        - objectName: "SECRET_2"
          key: "SECRET_2"

Apply the manifest:

kubectl apply -f secret-provider-class.yaml

Step 7: Deploy a Kubernetes Application

Create a deployment that mounts the secrets as volumes and exposes them as environment variables.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: csi-secret-app
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: csi-secret-app
  template:
    metadata:
      labels:
        app: csi-secret-app
    spec:
      serviceAccountName: demo-secrets-manager-service-account
      containers:
        - name: app-container
          image: nginx:latest
          env:
            - name: SECRET_1
              valueFrom:
                secretKeyRef:
                  name: demo-secrets
                  key: SECRET_1
            - name: SECRET_2
              valueFrom:
                secretKeyRef:
                  name: demo-secrets
                  key: SECRET_2
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: demo-secrets

Apply the manifest:

kubectl apply -f deployment.yaml

Step 8: Validate the Secrets Injection

Validate that the secrets have been properly mounted and synced as Kubernetes secrets.

Check if the Kubernetes secret exists:

kubectl get secret demo-secrets -n demo -o yaml

Validate the secrets within the application pod:

kubectl exec -n demo <pod-name> -- cat /mnt/secrets/SECRET_1
kubectl exec -n demo <pod-name> -- cat /mnt/secrets/SECRET_2

You should see the values of SECRET_1 and SECRET_2 correctly injected.

Conclusion

By leveraging the Secrets Store CSI Driver and AWS Secrets Manager, you can securely manage and inject secrets into Kubernetes workloads. This approach reduces the need for hardcoding sensitive data, ensures fine-grained IAM control, and integrates seamlessly with Kubernetes.

For more details, refer to the official documentation:

• AWS Secrets Store CSI Driver Integration
• Secrets Store CSI Driver Provider AWS
• Sync Kubernetes Secrets
• Secret Store CSI Driver Tutorial | Kubernetes Secrets | AWS Secrets Manager