> ## Documentation Index
> Fetch the complete documentation index at: https://kubestarterkit.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Secrets

> Create and manage secrets with External Secrets and AWS Secrets Manager

## Overview

The kit uses [External Secrets Operator](https://external-secrets.io/) to sync secrets from AWS Secrets Manager into Kubernetes. This keeps sensitive data out of Git while providing a GitOps-friendly secret management workflow.

## Architecture

```
AWS Secrets Manager          Kubernetes Cluster
┌───────────────────┐        ┌────────────────────────────┐
│                   │        │                            │
│ ksk-use2-staging- │        │  ClusterSecretStore        │
│   myapp-db        │<───────│  (aws-secrets-manager)     │
│                   │        │            │               │
└───────────────────┘        │            ▼               │
                             │  ExternalSecret            │
                             │  (myapp-secrets)           │
                             │            │               │
                             │            ▼               │
                             │  Secret                    │
                             │  (myapp-secrets)           │
                             │                            │
                             └────────────────────────────┘
```

1. **ClusterSecretStore** - Configures access to AWS Secrets Manager (one per cluster)
2. **ExternalSecret** - Defines which secrets to fetch and how to map them
3. **Secret** - The Kubernetes Secret created and kept in sync

## Prerequisites

The External Secrets Operator uses Pod Identity to authenticate to AWS. This is configured automatically by the EKS Terraform module.

Verify the ClusterSecretStore is working:

```bash theme={null}
kubectl get clustersecretstores
kubectl describe clustersecretstore aws-secrets-manager
```

## Create a Secret in AWS

<Steps>
  <Step title="Create the secret in Secrets Manager">
    Using AWS CLI:

    ```bash theme={null}
    export REGION="us-east-2"  # Your AWS region

    aws secretsmanager create-secret \
      --name "ksk-use2-staging-myapp-db" \
      --secret-string "postgres://user:pass@host:5432/db" \
      --region ${REGION}
    ```

    Or using the AWS Console:

    1. Navigate to Secrets Manager
    2. Click "Store a new secret"
    3. Choose "Other type of secret"
    4. Enter key/value pairs or plaintext
    5. Name it following the [naming convention](#secret-naming-conventions)
  </Step>

  <Step title="Create the ExternalSecret manifest">
    Create `ExternalSecret.myapp-secrets.yaml` in your service's templates:

    ```yaml theme={null}
    apiVersion: external-secrets.io/v1
    kind: ExternalSecret
    metadata:
      name: myapp-secrets
      namespace: myapp
    spec:
      refreshInterval: 1h
      secretStoreRef:
        name: aws-secrets-manager
        kind: ClusterSecretStore
      target:
        name: myapp-secrets
        creationPolicy: Owner
      data:
        - secretKey: DATABASE_URL
          remoteRef:
            key: ksk-use2-staging-myapp-db
    ```
  </Step>

  <Step title="Deploy and verify">
    ```bash theme={null}
    # Apply the ExternalSecret
    kubectl apply -f ExternalSecret.myapp-secrets.yaml

    # Check the ExternalSecret status
    kubectl get externalsecret myapp-secrets -n myapp

    # Verify the Secret was created
    kubectl get secret myapp-secrets -n myapp
    kubectl get secret myapp-secrets -n myapp -o jsonpath='{.data.DATABASE_URL}' | base64 -d
    ```
  </Step>
</Steps>

## Secret Naming Conventions

The kit uses [CloudPosse null-label](https://github.com/cloudposse/terraform-null-label) for consistent resource naming. Secrets follow the same pattern:

```
{namespace}-{environment}-{stage}-{name}
```

| Component     | Description               | Examples                                                |
| ------------- | ------------------------- | ------------------------------------------------------- |
| `namespace`   | Organization abbreviation | `ksk` (kube-starter-kit), `myco`, etc.                  |
| `environment` | AWS region abbreviation   | `use2` (us-east-2), `use1` (us-east-1), `gbl` (global)  |
| `stage`       | Deployment stage          | `staging`, `prod`, `shared`                             |
| `name`        | Descriptive secret name   | `myapp-db`, `argocd-github-dex`, `signoz-ingestion-key` |

<Note>
  The examples below use `ksk` as the namespace (the kit's default). Replace this with the namespace you configured during [Bootstrap Accounts](/usage/getting-started/03-bootstrap-accounts).
</Note>

Examples:

* `ksk-use2-staging-myapp-db` - Database credentials for myapp
* `ksk-use2-staging-argocd-github-dex` - ArgoCD GitHub OAuth credentials
* `ksk-use2-staging-signoz-ingestion-key` - SigNoz observability ingestion key

This convention:

* Aligns with all other Terraform-managed resources
* Makes secrets easy to identify by cluster/environment
* Enables fine-grained IAM policies using prefixes
* Avoids naming conflicts across environments

## Fetch Multiple Values from One Secret

AWS Secrets Manager secrets can contain JSON with multiple key/value pairs:

<Steps>
  <Step title="Create a JSON secret">
    ```bash theme={null}
    export REGION="us-east-2"  # Your AWS region

    aws secretsmanager create-secret \
      --name "ksk-use2-staging-myapp-config" \
      --secret-string '{"DB_HOST":"localhost","DB_USER":"admin","DB_PASS":"secret"}' \
      --region ${REGION}
    ```
  </Step>

  <Step title="Extract specific keys">
    ```yaml theme={null}
    apiVersion: external-secrets.io/v1
    kind: ExternalSecret
    metadata:
      name: myapp-config
      namespace: myapp
    spec:
      refreshInterval: 1h
      secretStoreRef:
        name: aws-secrets-manager
        kind: ClusterSecretStore
      target:
        name: myapp-config
      data:
        - secretKey: DB_HOST
          remoteRef:
            key: ksk-use2-staging-myapp-config
            property: DB_HOST
        - secretKey: DB_USER
          remoteRef:
            key: ksk-use2-staging-myapp-config
            property: DB_USER
        - secretKey: DB_PASS
          remoteRef:
            key: ksk-use2-staging-myapp-config
            property: DB_PASS
    ```
  </Step>
</Steps>

## Use Secrets in Pods

Reference the synced Secret in your Deployment:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp
          env:
            # Single environment variable
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: myapp-secrets
                  key: DATABASE_URL

            # Or load all keys as env vars
          envFrom:
            - secretRef:
                name: myapp-config
```

## Refresh and Sync Behavior

### Automatic Refresh

ExternalSecrets periodically refresh from AWS based on `refreshInterval`:

```yaml theme={null}
spec:
  refreshInterval: 1h  # Check for updates every hour
```

### Force Refresh

To immediately sync a secret:

```bash theme={null}
# Annotate to trigger refresh
kubectl annotate externalsecret myapp-secrets -n myapp force-sync=$(date +%s) --overwrite
```

### Reloader Integration

The kit includes [Reloader](https://github.com/stakater/Reloader), which automatically restarts pods when their Secrets change:

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    reloader.stakater.com/auto: "true"  # Restart on any Secret/ConfigMap change
```

Or for specific secrets:

```yaml theme={null}
metadata:
  annotations:
    secret.reloader.stakater.com/reload: "myapp-secrets"
```

## Environment-Specific Secrets

Use different secret paths for each environment:

```yaml theme={null}
# values.yaml (staging)
secrets:
  databaseUrlKey: ksk-use2-staging-myapp-db

# values.production.yaml
secrets:
  databaseUrlKey: ksk-use2-prod-myapp-db
```

Then template the ExternalSecret:

```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: {{ .Values.secrets.databaseUrlKey }}
```

## IAM Permissions

The External Secrets service account needs permission to read secrets. This is configured via Pod Identity in the EKS Terraform module.

By default, the External Secrets Pod Identity role has access to all secrets (`arn:aws:secretsmanager:*:*:secret:*`). To restrict access to specific prefixes, update the policy in `terraform/modules/eks/base-infra-resources.tf`:

```hcl theme={null}
module "external_secrets_pod_identity" {
  # ...
  # Replace region and namespace prefix with your values
  external_secrets_secrets_manager_arns = [
    "arn:aws:secretsmanager:<REGION>:*:secret:<namespace>-<env>-staging-*",
    "arn:aws:secretsmanager:<REGION>:*:secret:<namespace>-<env>-prod-*"
  ]
}
```

## Next Steps

* [Database Operations](/usage/operations/07-database-operations) - Manage database credentials
* [Bootstrapping a New Service](/usage/operations/04-bootstrapping-new-service) - Add secrets for new services