Managing Secrets

Overview

The kit uses External Secrets Operator to sync secrets from AWS Secrets Manager into Kubernetes. This keeps sensitive data out of Git while providing a GitOps-friendly secret management workflow.

Architecture

AWS Secrets Manager          Kubernetes Cluster
┌───────────────────┐        ┌────────────────────────────┐
│                   │        │                            │
│ ksk-use2-staging- │        │  ClusterSecretStore        │
│   myapp-db        │<───────│  (aws-secrets-manager)     │
│                   │        │            │               │
└───────────────────┘        │            ▼               │
                             │  ExternalSecret            │
                             │  (myapp-secrets)           │
                             │            │               │
                             │            ▼               │
                             │  Secret                    │
                             │  (myapp-secrets)           │
                             │                            │
                             └────────────────────────────┘

ClusterSecretStore - Configures access to AWS Secrets Manager (one per cluster)
ExternalSecret - Defines which secrets to fetch and how to map them
Secret - The Kubernetes Secret created and kept in sync

Prerequisites

The External Secrets Operator uses Pod Identity to authenticate to AWS. This is configured automatically by the EKS Terraform module. Verify the ClusterSecretStore is working:

kubectl get clustersecretstores
kubectl describe clustersecretstore aws-secrets-manager

Create a Secret in AWS

Create the secret in Secrets Manager

Using AWS CLI:

export REGION="us-east-2"  # Your AWS region

aws secretsmanager create-secret \
  --name "ksk-use2-staging-myapp-db" \
  --secret-string "postgres://user:pass@host:5432/db" \
  --region ${REGION}

Or using the AWS Console:

Navigate to Secrets Manager
Click “Store a new secret”
Choose “Other type of secret”
Enter key/value pairs or plaintext
Name it following the naming convention

Create the ExternalSecret manifest

Create ExternalSecret.myapp-secrets.yaml in your service’s templates:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: ksk-use2-staging-myapp-db

Deploy and verify

# Apply the ExternalSecret
kubectl apply -f ExternalSecret.myapp-secrets.yaml

# Check the ExternalSecret status
kubectl get externalsecret myapp-secrets -n myapp

# Verify the Secret was created
kubectl get secret myapp-secrets -n myapp
kubectl get secret myapp-secrets -n myapp -o jsonpath='{.data.DATABASE_URL}' | base64 -d

Secret Naming Conventions

The kit uses CloudPosse null-label for consistent resource naming. Secrets follow the same pattern:

{namespace}-{environment}-{stage}-{name}

Component	Description	Examples
`namespace`	Organization abbreviation	`ksk` (kube-starter-kit), `myco`, etc.
`environment`	AWS region abbreviation	`use2` (us-east-2), `use1` (us-east-1), `gbl` (global)
`stage`	Deployment stage	`staging`, `prod`, `shared`
`name`	Descriptive secret name	`myapp-db`, `argocd-github-dex`, `signoz-ingestion-key`

The examples below use ksk as the namespace (the kit’s default). Replace this with the namespace you configured during Bootstrap Accounts.

Examples:

ksk-use2-staging-myapp-db - Database credentials for myapp
ksk-use2-staging-argocd-github-dex - ArgoCD GitHub OAuth credentials
ksk-use2-staging-signoz-ingestion-key - SigNoz observability ingestion key

This convention:

Aligns with all other Terraform-managed resources
Makes secrets easy to identify by cluster/environment
Enables fine-grained IAM policies using prefixes
Avoids naming conflicts across environments

Fetch Multiple Values from One Secret

AWS Secrets Manager secrets can contain JSON with multiple key/value pairs:

Create a JSON secret

export REGION="us-east-2"  # Your AWS region

aws secretsmanager create-secret \
  --name "ksk-use2-staging-myapp-config" \
  --secret-string '{"DB_HOST":"localhost","DB_USER":"admin","DB_PASS":"secret"}' \
  --region ${REGION}

Extract specific keys

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-config
  namespace: myapp
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: myapp-config
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: ksk-use2-staging-myapp-config
        property: DB_HOST
    - secretKey: DB_USER
      remoteRef:
        key: ksk-use2-staging-myapp-config
        property: DB_USER
    - secretKey: DB_PASS
      remoteRef:
        key: ksk-use2-staging-myapp-config
        property: DB_PASS

Use Secrets in Pods

Reference the synced Secret in your Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp
          env:
            # Single environment variable
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myapp-secrets
                  key: DATABASE_URL

            # Or load all keys as env vars
          envFrom:
            - secretRef:
                name: myapp-config

Refresh and Sync Behavior

Automatic Refresh

ExternalSecrets periodically refresh from AWS based on refreshInterval:

spec:
  refreshInterval: 1h  # Check for updates every hour

Force Refresh

To immediately sync a secret:

# Annotate to trigger refresh
kubectl annotate externalsecret myapp-secrets -n myapp force-sync=$(date +%s) --overwrite

Reloader Integration

The kit includes Reloader, which automatically restarts pods when their Secrets change:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    reloader.stakater.com/auto: "true"  # Restart on any Secret/ConfigMap change

Or for specific secrets:

metadata:
  annotations:
    secret.reloader.stakater.com/reload: "myapp-secrets"

Environment-Specific Secrets

Use different secret paths for each environment:

# values.yaml (staging)
secrets:
  databaseUrlKey: ksk-use2-staging-myapp-db

# values.production.yaml
secrets:
  databaseUrlKey: ksk-use2-prod-myapp-db

Then template the ExternalSecret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: {{ .Values.secrets.databaseUrlKey }}

IAM Permissions

The External Secrets service account needs permission to read secrets. This is configured via Pod Identity in the EKS Terraform module. By default, the External Secrets Pod Identity role has access to all secrets (arn:aws:secretsmanager:*:*:secret:*). To restrict access to specific prefixes, update the policy in terraform/modules/eks/base-infra-resources.tf:

module "external_secrets_pod_identity" {
  # ...
  # Replace region and namespace prefix with your values
  external_secrets_secrets_manager_arns = [
    "arn:aws:secretsmanager:<REGION>:*:secret:<namespace>-<env>-staging-*",
    "arn:aws:secretsmanager:<REGION>:*:secret:<namespace>-<env>-prod-*"
  ]
}

Next Steps

Database Operations - Manage database credentials
Bootstrapping a New Service - Add secrets for new services

About

Usage

Overview

Architecture

Prerequisites

Create a Secret in AWS

Secret Naming Conventions

Fetch Multiple Values from One Secret

Use Secrets in Pods

Refresh and Sync Behavior

Automatic Refresh

Force Refresh

Reloader Integration

Environment-Specific Secrets

IAM Permissions

Next Steps

​Overview

​Architecture

​Prerequisites

​Create a Secret in AWS

​Secret Naming Conventions

​Fetch Multiple Values from One Secret

​Use Secrets in Pods

​Refresh and Sync Behavior

​Automatic Refresh

​Force Refresh

​Reloader Integration

​Environment-Specific Secrets

​IAM Permissions

​Next Steps

Overview

Architecture

Prerequisites

Create a Secret in AWS

Secret Naming Conventions

Fetch Multiple Values from One Secret

Use Secrets in Pods

Refresh and Sync Behavior

Automatic Refresh

Force Refresh

Reloader Integration

Environment-Specific Secrets

IAM Permissions

Next Steps