# About Me
Source: https://kubestarterkit.com/about-me
The person behind Kube Starter Kit
# About Me
I'm **Sid Palas**, the creator of Kube Starter Kit and founder of [DevOps Directive](https://devopsdirective.com/about/).
I've helped engineering teams at companies across industries build and scale their infrastructure. Kube Starter Kit distills the knowledge and patterns I've gathered over the past 6 years into a production-ready starting point for your Kubernetes platform.
## Companies I've Worked With
Healthcare data platform
DeFi risk management
Satellite imagery analytics
Aerospace data platform
## DevOps Training
I also create world-class DevOps training content to help engineers level up their infrastructure skills.
Free video tutorials and deep dives
Structured, in-depth learning paths
# Architecture Decisions
Source: https://kubestarterkit.com/architecture-decisions/index
Why I chose this technology stack
This page summarizes the key architectural decisions behind Kube Starter Kit. Each feature page includes detailed rationale in its "Key Design Decisions" section, this is the high-level overview of the philosophy and cross-cutting choices.
## Guiding Principles
**Opinionated but escapable:** I've made choices so you don't have to, but nothing locks you in. Every component can be swapped out as your needs evolve.
**Production-ready from day one:** This isn't a learning exercise or demo. It's infrastructure I'd run in production, with proper security, observability, and operational patterns.
**Balanced approach to managed services:** Kubernetes-native solutions where they add flexibility (ingress, observability, GitOps), managed services where the operational burden justifies it (object storage, databases, identity). The goal is portability without reinventing the wheel.
**GitOps everything:** Infrastructure, configuration, and application deployments all flow through Git. One source of truth, full audit trail.
## Technology Choices
| Category | Choice | Alternatives Considered |
| --------------------------- | -------------------------------------- | --------------------------------------------- |
| **Cloud Provider** | AWS | GCP, Azure |
| **Infrastructure as Code** | Terraform + Terramate | Pulumi, CDK, CloudFormation, Terragrunt |
| **Terraform Orchestration** | Terramate | Digger, Atlantis, Spacelift, Terraform Cloud |
| **Kubernetes** | EKS | Self-managed, GKE, AKS |
| **Node Provisioning** | Karpenter | Cluster Autoscaler |
| **GitOps** | ArgoCD | Flux, Kluctl |
| **Ingress** | Traefik | AWS ALB Ingress, ingress-nginx, Envoy Gateway |
| **Secrets** | External Secrets + AWS Secrets Manager | Sealed Secrets, Vault |
| **Observability** | SigNoz | Datadog, Grafana Stack |
| **Database** | PlanetScale | RDS, CloudNativePG |
| **Local Development** | KinD + Tilt + mirrord | Docker Compose, Skaffold, Telepresence |
## Why These Choices?
### Cloud: AWS
AWS has the largest market share and the best ecosystem of supporting services (S3, RDS, SQS, etc.). Most third-party infrastructure tooling runs on AWS, so co-locating your workloads enables reduced latency and the potential for private networking (more secure and more affordable). Most engineers have AWS experience. For early-stage companies, AWS is the safe choice.
### Infrastructure as Code: Terraform
Terraform has the largest community, most modules, and best tooling ecosystem. Pulumi and CDK are interesting but add complexity and have smaller ecosystems.
### Orchestration: Terramate
Terramate provides a unified solution for Terraform orchestration: stack management, code generation, outputs sharing, and CI/CD integration with Terramate Cloud. Unlike Terragrunt, it uses native HCL syntax without a wrapper. Unlike Digger or Atlantis, it includes change detection and cross-stack dependency management. Terramate Cloud provides visibility into previews, deployments, and drift, without requiring external services to access your AWS credentials.
### GitOps: ArgoCD
ArgoCD, Flux, and Kluctl are all solid choices. ArgoCD has the best UI for understanding deployment state, which helps teams new to GitOps. The app-of-apps pattern provides clear hierarchical organization. Flux is lighter-weight but less visual. Kluctl offers a more imperative approach with better diffing and templating flexibility, worth considering if you prefer CLI-driven workflows.
### Ingress: Traefik
Traefik is one of the most popular ingress solutions (3B+ downloads, 57K GitHub stars). It supports both traditional Ingress and Gateway API, letting you start simple and adopt advanced routing as needs evolve. Single binary, automatic discovery, sensible defaults, built-in dashboard.
Initially I used ingress-nginx as the default ingress controller for the kit, but it is being retired in 2026, with no updates/maintenance.
### Secrets: External Secrets
Secrets belong in a secrets manager, not encrypted in Git. External Secrets keeps secrets in AWS Secrets Manager where they can be rotated, audited, and access-controlled properly. Vault is powerful but adds significant operational complexity (self-hosted) or cost (HCP Vault). For most teams, AWS Secrets Manager with External Secrets hits the sweet spot.
### Observability: SigNoz
For early-stage companies, Datadog's per-host pricing gets expensive fast. SigNoz gives you metrics, logs, and traces with clear and affordable pricing with the option to self-host down the line as needed.
### Local Development: KinD + Tilt + mirrord
KinD provides a real multi-node Kubernetes cluster locally. Tilt handles continuous rebuilds and deploys with smart dependency ordering. mirrord enables debugging local code against remote cluster traffic without deploying. Together they cover the full development lifecycle: local iteration, integration testing, and production debugging.
## What's Not Included (Yet)
Some things are intentionally omitted from the initial kit:
* **Service mesh:** Most teams don't need Istio/Linkerd on day one. Add it when you have specific requirements.
* **Multi-region:** Single region is simpler to operate. The patterns here extend to multi-region when you need it.
* **Advanced networking:** No VPC peering, Transit Gateway, or PrivateLink beyond basics. Add as needed.
These are candidates for potential extensions as the kit evolves.
# Terraform for Base Infrastructure
Source: https://kubestarterkit.com/features/01-terraform
Modular, well-structured Terraform for AWS with isolated state and battle-tested modules
## The Problem
Infrastructure as Code is table stakes for any serious engineering team, but getting it *right* is surprisingly hard:
* **Sprawling, copy-pasted modules:** Teams often start with one environment and end up with duplicated Terraform code everywhere, each copy drifting slightly from the others.
* **State management headaches:** Where do you store state? How do you handle locking? How do you structure state files so changes in one area don't require touching unrelated infrastructure?
* **No clear patterns for multi-environment:** Development, staging, production... how do you manage the differences without maintaining three separate codebases?
* **Reinventing the wheel:** Writing VPC, EKS, and IAM configurations from scratch means debugging problems that others have already solved.
## How Kube Starter Kit Addresses This
### Isolated State Per Stack
Each deployable unit (networking, EKS cluster, app-resources) has its own state file. This means you can update your EKS cluster without Terraform needing to refresh your entire VPC state. A bad apply in one area doesn't risk corrupting unrelated infrastructure.
### Battle-Tested Community Modules
Rather than maintaining VPC and EKS code from scratch, I build on top of well-maintained modules from the [Terraform AWS Modules](https://github.com/terraform-aws-modules) project. These are used by thousands of teams and handle edge cases you'd otherwise discover the hard way. The kit adds the glue and opinions that make them work together.
### Hierarchical Configuration
Settings cascade from root to environment to region to stack. Common values like your namespace prefix, provider versions, and IAM roles are defined once and inherited everywhere. Each environment only defines what's unique: region, sizing, feature flags.
### Application Resources Pattern
Per-application AWS resources (IAM roles for Pod Identity, Secrets Manager entries, database credentials) are provisioned alongside infrastructure but in separate stacks. This keeps application concerns isolated while maintaining the same workflow for all Terraform changes.
## What's Provisioned
One-time setup for new AWS accounts:
* S3 bucket for Terraform state with native locking (no DynamoDB needed)
* GitHub OIDC provider for keyless CI/CD authentication
* IAM roles for Terraform automation
Production-ready VPC with:
* Public and private subnets across 3 availability zones
* Configurable NAT gateway options (single, per-AZ, or fck-nat for cost savings)
* Proper subnet tagging for Karpenter node discovery
* VPC endpoint support for private AWS service access
Fully-configured EKS cluster with:
* Managed node group for baseline capacity (runs Karpenter itself)
* Essential add-ons pre-configured (CoreDNS, VPC CNI, EBS CSI driver, Pod Identity)
* IAM integration via AWS SSO for cluster access
* Security group rules for proper inter-node communication
Per-application resources provisioned via dedicated stacks:
* IAM roles for Pod Identity (AWS access from Kubernetes pods)
* Secrets Manager entries for application secrets
* Database credentials and connection strings
* Any other AWS or third-party resources specific to an application
These resources are referenced by Kubernetes deployments via External Secrets.
GitHub and AWS IAM Identity Center from a single source of truth:
* GitHub organization membership and team assignments
* AWS SSO users and group memberships
* Permission sets mapped to AWS accounts
See [User Management](/features/04-user-management) for details.
## Directory Structure
```
terraform/
├── bootstrap/ # One-time account setup (state bucket, OIDC)
├── modules/ # Reusable Terraform modules
│ ├── eks/
│ ├── networking/
│ └── app-resources/
└── live/ # Stack definitions by environment
├── shared/ # Cross-account resources
│ ├── global/ # IAM Identity Center, GitHub org management
│ └── {region}/ # ECR repositories
└── {stage}/ # staging, prod, etc.
├── global/ # Account-level resources (bootstrapping, DNS)
└── {region}/ # us-east-1, us-east-2, etc.
├── networking/
├── eks/
└── app-resources/
```
## Key Design Decisions
| Decision | Rationale |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Terraform over Pulumi/CDK** | Declarative HCL is easier to review and reason about than imperative code. Larger ecosystem of modules and community support. |
| **Terraform over Crossplane** | Simpler operational model with no controller running in-cluster to manage. Better tooling for plan/preview workflows. Crossplane shines for self-service platforms, but adds complexity for smaller teams. |
| **One state file per stack** | Blast radius containment. A networking change shouldn't require EKS state refresh. Failures are isolated. |
| **Community modules as building blocks** | The terraform-aws-modules are battle-tested by thousands of users. Layer opinions on top rather than maintaining infrastructure code from scratch. |
| **S3 native locking** | Simpler than DynamoDB locking, fewer moving parts (requires Terraform 1.10+ or OpenTofu 1.8+). |
| **Hierarchical config over copy-paste** | Define common values once, override only what differs per environment. Changes propagate automatically. |
For orchestration across stacks (change detection, dependency ordering, CI/CD integration), see [Terraform Orchestration with Terramate](/features/02-terramate).
For making changes to infrastructure, see [Making Terraform Changes](/usage/operations/01-updating-terraform-infrastructure).
# Terraform Orchestration with Terramate
Source: https://kubestarterkit.com/features/02-terramate
Change detection, dependency ordering, and CI/CD integration for multi-stack Terraform
## The Problem
Running Terraform locally works fine when you're a team of one, but it quickly becomes a liability as you grow:
* **"Who ran that apply?":** Without centralized execution, it's hard to track who changed what and when. Your state file says it changed, but good luck figuring out the context.
* **No visibility before merge:** You want to review infrastructure changes before they happen, but `terraform plan` output buried in a CI log isn't exactly reviewer-friendly.
* **Credential sprawl:** Every developer with Terraform access needs AWS credentials. That's a lot of long-lived secrets floating around laptops.
* **Cross-stack dependencies are painful:** `terraform_remote_state` data sources are clunky and require knowing state bucket details everywhere.
* **Running all stacks is slow:** Without change detection, every PR plans every stack, even unchanged ones.
## How Kube Starter Kit Addresses This
I've integrated [Terramate](https://terramate.io/) to orchestrate Terraform across environments. Here's what that gives you:
### Change Detection
Terramate detects which stacks are affected by your changes. On a PR, only modified stacks get planned, not your entire infrastructure. This makes CI faster, reduces noise, and lowers costs.
### Dependency Ordering
Stacks declare dependencies and Terramate runs them in the correct order. Networking before EKS, EKS before app-resources. No manual coordination required.
### Outputs Sharing
Stacks can consume outputs from other stacks without `terraform_remote_state`. Define an output in one stack, consume it as a variable in another. Terramate handles the wiring. Dependencies are explicit, type-safe, and support mocks for bootstrapping.
### Code Generation
Common patterns (backend config, provider setup, module invocations) are generated from templates called "mixins." Change a mixin once, regenerate everywhere. This eliminates copy-paste drift between stacks.
### Terramate Cloud
Previews, deployments, and drift detection sync to a dashboard. See the state of your infrastructure across all stacks in one place: who requested changes, who approved them, what actually changed.
### Keyless Authentication
GitHub OIDC assumes an AWS role. No long-lived credentials stored in GitHub secrets or on developer laptops. The trust is based on GitHub's identity, not shared secrets.
## The Workflow
Make changes to Terraform code: modules, stacks, or configuration.
Terramate detects affected stacks and runs `terraform plan` for each. Results sync to Terramate Cloud and appear in the PR checks.
Reviewers see exactly what infrastructure changes will happen. Check the Terramate Cloud dashboard for a unified view across stacks.
Merge the PR to main when approved.
The deploy workflow applies changed stacks in dependency order. Results sync to Terramate Cloud.
Every change is tied to a PR and tracked in Terramate Cloud: who requested it, who approved it, what changed.
## CI/CD Workflows
Four GitHub Actions workflows handle different scenarios:
| Workflow | Trigger | Purpose |
| ------------------- | ----------------- | ------------------------------------------------------- |
| **Preview** | Pull request | Plans changed stacks, syncs previews to Terramate Cloud |
| **Deploy** | Merge to main | Applies changed stacks in dependency order |
| **Drift Detection** | Schedule/manual | Detects when infrastructure has drifted from state |
| **Provider Cache** | Lock file changes | Pre-downloads providers to speed up other workflows |
## Key Design Decisions
| Decision | Rationale |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Terramate over Digger/Atlantis** | Terramate provides stack orchestration, code generation, and outputs sharing, not just CI/CD. The unified platform reduces tooling complexity. |
| **Apply on merge (not before)** | Simpler workflow: merge triggers apply. Terramate Cloud provides visibility if rollback is needed. |
| **Outputs sharing over remote\_state** | No need to pass bucket names everywhere. Dependencies are explicit and type-safe. Mocks enable planning before dependencies exist. |
| **OIDC over static credentials** | No secrets to rotate. Short-lived tokens exchanged at runtime. |
| **Change detection by default** | Only plan/apply what changed. Faster CI, less noise, lower costs. |
For the infrastructure that Terramate orchestrates, see [Terraform for Base Infrastructure](/features/01-terraform).
For step-by-step instructions on making changes, see [Making Terraform Changes](/usage/operations/01-updating-terraform-infrastructure).
# AWS Architecture
Source: https://kubestarterkit.com/features/03-aws-architecture
Multi-account setup with VPC, EKS, and proper IAM boundaries
## The Problem
AWS gives you a blank canvas, which is both a blessing and a curse:
* **Single-account sprawl:** Many teams start with one AWS account and end up with staging and production resources tangled together. One bad IAM policy or accidental deletion affects everything.
* **Network design paralysis:** How many AZs? Public subnets, private subnets, or both? NAT Gateway (expensive) or something else? These decisions are hard to change later.
* **EKS configuration complexity:** Getting EKS right involves dozens of settings: add-ons, IAM roles for service accounts, node groups vs Karpenter, security groups, access management...
* **Cost surprises:** NAT Gateway charges, idle node capacity, and over-provisioned resources add up fast. By the time you notice, you've burned through budget.
## How Kube Starter Kit Addresses This
I've designed the AWS architecture in this kit based on patterns that balance security, cost, and operational simplicity:
**Multi-account by default:** Staging and production live in separate AWS accounts. This provides hard isolation: a mistake in staging can't affect production resources, IAM policies are naturally scoped, and billing is automatically separated.
**VPC with flexibility:** Each environment gets a VPC spanning 3 availability zones with public and private subnets. The NAT configuration is pluggable: use AWS NAT Gateway for production reliability, or [fck-nat](https://fck-nat.dev/) for non-production cost savings.
**EKS with sensible defaults:** The cluster comes pre-configured with essential add-ons (CoreDNS, VPC CNI, EBS CSI driver), proper IAM integration via AWS SSO, and a base node group sized for running Karpenter and critical workloads.
**Karpenter for right-sized compute:** Instead of managing multiple node groups for different workload types, Karpenter provisions exactly the nodes you need, when you need them. Less waste, less management.
## What's Included
### Account Structure
```
AWS Organization
├── Management Account
│ ├── AWS Organizations
│ ├── IAM Identity Center (Manages user access across all accounts)
│ └── Consolidated Billing
├── Infrastructure Account
│ ├── GitHub OIDC Provider
│ ├── Terraform State Bucket
│ └── IAM Roles for Terraform Automation (Manages all account infrastructure)
├── ECR Account
│ └── ECR Repositories (Consumed by staging + production)
├── Staging Account
│ ├── VPC
│ ├── EKS Cluster
│ ├── Secrets Manager
│ ├── Route 53 Hosted Zone
│ └── Application-specific Resources (IAM roles, S3, etc.)
└── Production Account
├── VPC
├── EKS Cluster
├── Secrets Manager
├── Route 53 Hosted Zone
└── Application-specific Resources (IAM roles, S3, etc.)
```
![AWS Architecture Diagram]()
### VPC Architecture
Each environment VPC includes:
| Component | Configuration |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| **Availability Zones** | 3 AZs for high availability |
| **Public Subnets** | For load balancers and NAT |
| **Private Subnets** | For EKS nodes and workloads |
| **NAT** | Configurable: [AWS NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) or [fck-nat](https://fck-nat.dev/) |
| **Subnet Tagging** | Pre-configured for Karpenter discovery |
| **Bastion Host** | Optional EC2 instance for private EKS API access via SSM |
### EKS Cluster Configuration
Pre-configured and version-pinned:
* **CoreDNS:** Cluster DNS
* **VPC CNI:** Pod networking with native AWS IPs
* **kube-proxy:** Service networking
* **EBS CSI Driver:** Persistent volume support
* **Pod Identity Agent:** Modern pod-level IAM
* **Base node group:** 2-3 nodes running Karpenter and critical infrastructure
* **Karpenter:** Provisions workload nodes on-demand with right-sized instances
* **ARM64 support:** Graviton instances for cost savings where compatible
* **AWS SSO integration:** Cluster access via IAM Identity Center
* **IRSA enabled:** Pods can assume IAM roles without node-level credentials
* **Pod Identity:** Newer, simpler alternative to IRSA for pod-level AWS access
### Cost Optimization
The kit includes several cost-conscious choices:
* **fck-nat for non-production:** NAT Gateway costs `~$32/month` per AZ just to exist PLUS `$0.045/GB` processed. fck-nat runs on a `t4g.nano (~$3/month)` and handles typical dev/staging traffic fine.
* **Karpenter consolidation:** Automatically bins packs workloads and removes underutilized nodes.
* **Spot instances:** Karpenter can provision spot instances for fault-tolerant workloads.
* **Right-sized base nodes:** The base node group uses smaller instances since it only runs infrastructure components.
## Key Design Decisions
| Decision | Rationale |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Multi-account over single account** | Hard isolation between environments. Can't accidentally affect production from staging. Cleaner IAM boundaries. |
| **3 AZs** | Balance between availability and cost. 2 AZs risk losing half capacity; more than 3 adds cost without proportional benefit. |
| **Private subnets for nodes** | Nodes don't need public IPs. Reduces attack surface and simplifies security groups. |
| **Karpenter over Cluster Autoscaler** | Faster scaling, better bin-packing, supports mixed instance types without managing multiple node groups. |
| **EKS managed add-ons** | AWS handles upgrades and compatibility. Less operational burden than self-managed. |
# GitHub + AWS User Management
Source: https://kubestarterkit.com/features/04-user-management
Unified identity management across GitHub and AWS from a single source of truth
## The Problem
User management across multiple systems is a constant source of friction and security risk:
* **Multiple sources of truth:** GitHub org membership in one place, AWS IAM users in another, SSO groups somewhere else. When someone joins or leaves, you're updating multiple systems.
* **Permission drift:** Over time, permissions accumulate. Temporary access grants become permanent without regular audits.
* **Audit complexity:** Answering "who has access to production?" requires checking multiple systems and correlating information manually.
* **Onboarding/offboarding overhead:** New hires need accounts created in multiple places. Departures require remembering all the places to revoke access.
## How Kube Starter Kit Addresses This
I've built a unified user management system where a single YAML file drives both GitHub and AWS access:
**Single source of truth:** One `users.yaml` file defines who has access to what. GitHub org membership, team assignments, AWS SSO users, and group memberships all flow from this one file.
**Declarative and auditable:** Because it's in version control, you have a complete history of access changes. Who added that user? Check the git log. What permissions did they have last month? Check the commit history.
**Consistent group model:** The same logical groups (Admin, ReadOnly, etc.) map to both GitHub teams and AWS SSO groups by default. This keeps things simple initially, but the model is extensible to any set of permissions across platforms as your needs evolve.
**PR-based access changes:** Adding a user or changing permissions is a pull request. It gets reviewed, approved, and automatically applied via Terraform.
## What's Included
### User Definition Format
Users are defined in a single YAML file with both GitHub and AWS configuration:
```yaml theme={null}
users:
- github:
username: jsmith
role: member # GitHub org role: admin or member
teams:
Admin:
role: maintainer # GitHub team role: maintainer or member
aws:
user_name: jane.smith
email: jane@company.com
group_membership: [Admin]
given_name: Jane
family_name: Smith
```
### GitHub Resources
The Terraform configuration manages:
| Resource | Description |
| -------------------------- | --------------------------------------------- |
| **Org membership** | Adds users to your GitHub organization |
| **Teams** | Creates teams (Admin, ReadOnly, etc.) |
| **Team membership** | Assigns users to teams with appropriate roles |
| **Repository permissions** | Grants teams access to the repository |
### AWS IAM Identity Center
The configuration creates:
Creates users in IAM Identity Center with email, name, and group assignments.
Pre-configured groups that map to permission levels:
* **Admin:** Full administrative access
* **PowerUser:** Developer access without IAM management
* **ReadOnly:** View-only access for auditing
AWS-managed policies attached to groups:
* **AdministratorAccess:** Full AWS access
* **PowerUserAccess:** Full access except IAM
* **ViewOnlyAccess:** Read-only across services
Groups are assigned to permission sets across all accounts (staging, production, shared, etc.).
### Workflow
Add a new user or modify permissions in the YAML file.
Terramate detects the changed stacks and runs `terraform plan`, showing exactly what will change.
Team members review the access change. Clear diff of who's getting what.
Changes are applied to both GitHub and AWS automatically.
### Example: Adding a New Team Member
```yaml theme={null}
# Add to users.yaml
- github:
username: newdev
role: member
teams:
Admin:
role: member
aws:
user_name: new.developer
email: new@company.com
group_membership: [Admin]
given_name: New
family_name: Developer
```
This single addition:
* Invites `newdev` to your GitHub organization
* Adds them to the Admin team with member permissions
* Creates an AWS SSO user
* Adds them to the Admin group with AdministratorAccess across all accounts
## Key Design Decisions
| Decision | Rationale |
| -------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **YAML over UI** | Version control, audit history, and code review for access changes. |
| **IAM Identity Center over IAM users** | Centralized identity, no long-lived credentials, integrates with external IdPs if needed later. |
| **Matching group names** | Cognitive simplicity: "Admin" means the same thing in GitHub and AWS. |
| **PR-based workflow** | Access changes get the same review process as code changes. No shadow IT. |
# GitOps with ArgoCD
Source: https://kubestarterkit.com/features/05-gitops
Declarative, auditable Kubernetes deployments driven by Git
## The Problem
Deploying to Kubernetes without GitOps often means:
* **Imperative chaos:** `kubectl apply` from laptops, scripts that "usually work," and no clear record of what's actually running.
* **Drift:** Someone makes a quick fix directly in the cluster. Now the running state doesn't match any source of truth.
* **No audit trail:** "Who deployed that change?" requires digging through CI logs, Slack messages, and hoping someone remembers.
* **Environment inconsistency:** Staging and production diverge over time because deployments are manual and error-prone.
* **Rollback friction:** Something breaks in production. How do you get back to the previous state? Which commit was that?
## How Kube Starter Kit Addresses This
I've implemented GitOps with ArgoCD, where Git is the single source of truth for your cluster state:
**Git as the source of truth:** Everything running in the cluster is defined in Git. If it's not in the repo, it shouldn't be in the cluster.
**Continuous reconciliation:** ArgoCD watches your Git repository and automatically syncs the cluster to match. Drift gets corrected automatically.
**App of Apps pattern:** A hierarchical structure where one ArgoCD Application manages other Applications. Add a new service by adding a file; ArgoCD picks it up automatically.
**Environment separation:** Each cluster (staging, production) has its own rendered manifests. Same source templates, different configurations.
**Audit everything:** Every deployment is a Git commit. Who changed what, when, and why is all in the commit history.
## What's Included
### ArgoCD Structure
```
kubernetes/
├── src/ # Source templates
│ ├── argocd/
│ │ ├── argocd/ # ArgoCD itself (bootstraps everything)
│ │ ├── infrastructure/ # Infrastructure apps definition
│ │ └── services/ # Application services definition
│ ├── infrastructure/ # Infrastructure components
│ │ ├── cert-manager/
│ │ ├── external-dns/
│ │ ├── traefik/
│ │ └── ...
│ └── services/ # Your applications
│ └── go-backend/
└── rendered/ # Environment-specific output
├── staging/
└── production/
```
### App of Apps Hierarchy
ArgoCD uses a three-tier Application structure:
```
argocd-app-of-apps (root)
├── argocd-app-of-apps (manages itself 🔄)
├── infrastructure-app-of-apps
│ ├── cert-manager
│ ├── external-dns
│ ├── traefik
│ ├── external-secrets
│ ├── karpenter
│ └── ...
└── services-app-of-apps
├── go-backend
└── ...
```
The root Application manages itself and the infrastructure and services Applications, which in turn manage individual components. This creates a clean separation between platform infrastructure and application workloads, while ensuring the entire ArgoCD configuration is also GitOps-managed.
### Project Organization
ArgoCD Projects provide logical separation:
| Project | Purpose |
| ------------------ | ------------------------------------------------- |
| **argocd** | ArgoCD's own configuration and app-of-apps |
| **infrastructure** | Platform components (cert-manager, ingress, etc.) |
| **services** | Application workloads |
### Deployment Flow
Update Kubernetes manifests or Helm values in the `src/` directory.
Run `mise run render-cluster ` from the application directory to generate environment-specific manifests to `rendered/staging/` or `rendered/production/`.
CI validates that rendered manifests are up-to-date (no git diff after re-rendering).
ArgoCD notices the rendered manifests differ from cluster state.
ArgoCD applies the changes to the cluster. With auto-sync enabled, this happens automatically.
ArgoCD reports sync status and health. Failed deployments are visible immediately.
### Configuration Management
Applications are enabled/disabled via a simple values file:
```yaml theme={null}
# infrastructure/values.yaml
applications:
cert-manager:
enabled: true
external-dns:
enabled: true
traefik:
enabled: true
karpenter:
enabled: true
envoy-gateway:
enabled: false # Not using this yet
```
To add a new infrastructure component, create its directory in `src/infrastructure/` and add an entry to the values file. ArgoCD picks it up on the next sync.
### Repository Access
ArgoCD authenticates to your Git repository using SSH deploy keys. During bootstrap, a mise task generates a key pair and creates a Kubernetes Secret with the private key. You add the public key to your GitHub repository as a deploy key.
No credentials in Git. ArgoCD discovers the repository secret automatically via the `argocd.argoproj.io/secret-type=repository` label. See [cluster bootstrap](/usage/getting-started/08-deploy-kubernetes-baseline#create-deploy-key) for setup instructions.
## Key Design Decisions
| Decision | Rationale |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| **ArgoCD over Flux** | Better UI for visibility, more mature ecosystem, easier onboarding for teams new to GitOps. |
| **App of Apps over ApplicationSets** | More explicit control over the hierarchy. ApplicationSets are powerful but can be harder to reason about. |
| **Rendered manifests in Git** | What's deployed is exactly what's in Git. No runtime templating surprises. Easier to audit and diff. |
| **Bootstrap-generated deploy keys** | Deploy keys are generated during bootstrap and stored as Kubernetes Secrets. Simple, no external dependencies. |
| **Auto-sync enabled** | Reduces manual toil. If you commit it, it deploys. Rollback is just reverting a commit. |
# Baseline Kubernetes Components
Source: https://kubestarterkit.com/features/06-k8s-baseline
Curated infrastructure components every production cluster needs
## The Problem
A fresh Kubernetes cluster is surprisingly bare. Out of the box, you can't:
* **Expose services to the internet:** No ingress/gateway api controller, no way to route external traffic to your pods.
* **Provision TLS certificates:** No automated certificate management. Manual cert provisioning is tedious and error-prone.
* **Manage DNS:** Creating an Ingress doesn't create public DNS records. That's a separate manual step.
* **Handle secrets properly:** Kubernetes Secrets are base64-encoded, not encrypted. Storing them in Git is a security risk.
* **Scale nodes automatically:** Pods can scale, but if there's no node capacity, they just sit Pending.
* **Observe what's happening:** No metrics, no logs aggregation, no tracing out of the box.
Every team ends up installing the same set of tools, making the same configuration decisions, and debugging the same integration issues.
## How Kube Starter Kit Addresses This
I've curated a set of infrastructure components that I've used across multiple production clusters. They're pre-configured to work together, with sensible defaults and AWS integrations already wired up:
**Ingress with automatic TLS:** Traefik handles traffic routing; cert-manager automatically provisions and renews Let's Encrypt certificates.
**DNS automation:** External-DNS watches your Ingress resources and creates Route53 records automatically. No more manually creating DNS entries.
**Secrets from AWS:** External Secrets syncs secrets from AWS Secrets Manager into Kubernetes. Secrets stay in a proper secrets manager, not in Git.
**Right-sized node provisioning:** Karpenter provisions exactly the nodes your workloads need, when they need them. No over-provisioning, no waiting for scale-up.
**Observability ready:** SigNoz provides metrics, logs, and traces in a single platform. See what's happening across your entire stack.
## What's Included
### Component Overview
| Component | Purpose | AWS Integration |
| -------------------- | ------------------------------------- | ---------------------------------- |
| **traefik** | HTTP(S) traffic routing | Creates NLB via AWS Load Balancer |
| **cert-manager** | Automatic TLS certificates | Uses Route53 for DNS-01 challenges |
| **external-dns** | DNS record management | Creates Route53 records |
| **external-secrets** | Secrets synchronization | Reads from AWS Secrets Manager |
| **karpenter** | Node auto-provisioning | Launches EC2 instances |
| **cloudnative-pg** | PostgreSQL operator | Uses EBS for persistent storage |
| **reloader** | Config/secret change detection | Restarts pods on updates |
| **SigNoz** | Observability (metrics, logs, traces) | - |
### Traffic Flow
```
Internet
│
▼
┌───────────────────────────────┐
│ AWS NLB │
│ (created by traefik) │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ traefik controller │
│ - TLS termination │
│ - Route by Ingress rules │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ Your application pods │
└───────────────────────────────┘
```
### Secrets Flow
```
AWS Secrets Manager
▲
│ (polls)
│
┌───────────────────────────────┐
│ ClusterSecretStore │
└───────────────────────────────┘
▲
│ (references)
│
┌───────────────────────────────┐
│ ExternalSecret │
└───────────────────────────────┘
│
│ (creates/updates)
▼
┌───────────────────────────────┐
│ Kubernetes Secret │
└───────────────────────────────┘
```
### Component Details
A modern cloud-native ingress controller that supports both Kubernetes Ingress and Gateway API. Deployed with:
* AWS NLB for external traffic (Layer 4)
* Automatic service discovery
* Native Kubernetes Ingress support
* Full Gateway API support (HTTPRoute, GRPCRoute, TCPRoute)
* IngressRoute CRDs for advanced routing
Using a single controller for both Ingress and Gateway API reduces operational complexity. Start with familiar Ingress resources today, migrate to Gateway API when ready, all without changing infrastructure.
Create an Ingress resource, and traffic flows automatically.
Automated X.509 certificate management. Configured with:
* Let's Encrypt production and staging issuers
* DNS-01 challenge solver via Route53
* Self-signed issuer for internal certificates
Add a `tls` section to your Ingress; cert-manager handles the rest.
Synchronizes Kubernetes resources with DNS providers. Configured to:
* Watch Ingress and Service resources
* Create/update Route53 records
* Use txt-registry to track ownership
Create an Ingress with a hostname; DNS record appears automatically.
Syncs external secrets into Kubernetes. Set up with:
* ClusterSecretStore pointing to AWS Secrets Manager
* IRSA authentication (no credentials in cluster)
* Automatic refresh on secret changes
Reference a secret in AWS; it appears as a Kubernetes Secret.
Just-in-time node provisioning. Configured with:
* NodePool defining instance requirements
* EC2NodeClass for AWS-specific settings
* Consolidation to remove underutilized nodes
* Support for spot instances
Deploy a pod that doesn't fit; Karpenter provisions a node.
PostgreSQL operator for Kubernetes. Provides:
* Declarative PostgreSQL clusters
* Automated backups to S3
* High availability with automatic failover
* Connection pooling via PgBouncer
Define a Cluster resource; get a production PostgreSQL.
Watches ConfigMaps and Secrets, restarts dependent pods on changes. Handles:
* Automatic pod rollouts when configs change
* Annotation-based opt-in per Deployment
Update a ConfigMap; pods restart automatically.
Open-source observability platform. Collects:
* Metrics from Kubernetes and applications
* Logs from all pods
* Distributed traces (OpenTelemetry)
Single pane of glass for your entire stack.
### Example: Exposing an Application
With all components working together, exposing an app is simple:
```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: traefik
tls:
- hosts:
- app.example.com
secretName: my-app-tls
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app
port:
number: 80
```
This single resource triggers:
1. **external-dns** creates `app.example.com` → NLB in Route53
2. **cert-manager** provisions a Let's Encrypt certificate
3. **traefik** routes traffic to your service
## Key Design Decisions
| Decision | Rationale |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Traefik over AWS ALB Ingress** | More portable, cloud-native design, excellent Kubernetes integration. Supports both Ingress and Gateway API in a single controller, reducing operational overhead. NLB at L4 is simpler. |
| **external-secrets over Sealed Secrets** | Secrets stay in a proper secrets manager. Better for rotation, auditing, and access control. |
| **Karpenter over Cluster Autoscaler** | Faster provisioning, better bin-packing, simpler configuration for mixed instance types. |
| **SigNoz over Datadog/New Relic** | Open source, self-hosted, no per-host pricing. Good enough for most teams starting out. |
| **CloudNativePG over RDS** | Runs in-cluster for lower latency and cost. Easy to move to RDS later if needed. |
# CI/CD Pipelines
Source: https://kubestarterkit.com/features/07-ci-cd-pipeline
Automated container builds and staging deployments on every merge to main
## Overview
The CI/CD pipeline automates container builds and deployments via GitHub Actions:
* **Change detection:** Only builds applications that have changed, keeping CI fast in a monorepo
* **Consistent builds:** All containers are built in CI with the same environment and tooling
* **Rendered manifests:** Kubernetes manifests are rendered and committed to Git, so what's in the repository is exactly what ArgoCD deploys
* **Automatic staging deployment:** Merges to main trigger builds and deploy to staging automatically
## What's Included
### Workflow Structure
```
.github/workflows/
├── ci-build-push-containers.yml # Build and push container images
├── ci-test.yml # Run tests (placeholder)
├── gitops-update-manifests.yml # Render and commit K8s manifests
└── terramate-*.yml # Terraform orchestration
```
Terraform CI/CD workflows (preview, deploy, drift detection) are covered in [Terraform Orchestration](/features/02-terramate#github-actions-workflows).
### Change Detection
The workflow uses [dorny/paths-filter](https://github.com/dorny/paths-filter) with a separate filter configuration file:
```yaml theme={null}
# .github/utils/file-filters.yaml
services/go-backend:
- "services/go-backend/**"
- "kubernetes/src/services/go-backend/**"
services/another-service:
- "services/another-service/**"
- "kubernetes/src/services/another-service/**"
```
The workflow matrix builds only the services that match changed paths. Changes to shared dependencies (like the workflow itself) trigger rebuilds of all dependent services.
### Build and Deploy Process
Determines which services have modifications based on path filters.
Builds and pushes a container image to ECR for each changed service.
Triggers `gitops-update-manifests` for each built service.
Generates Kubernetes manifests with the new image tags.
Pushes the updated manifests to the repository.
Detects the manifest changes via webhook or polling and deploys automatically.
### Container Registry
Images are pushed to Amazon ECR with version tags derived from `git describe`:
```
.dkr.ecr..amazonaws.com/go-backend:1.2.3-0001-g4b5d2e7
```
The tag format `--g` shows the base version from the last release tag, how many commits since that release, and the commit hash. This provides traceability, immutability (tags are never overwritten), and easy rollback.
### Manifest Rendering
The pipeline renders Helm charts or Kustomize overlays into plain Kubernetes manifests:
```
kubernetes/
├── src/
│ └── services/
│ └── go-backend/
│ ├── Chart.yaml
│ └── values.yaml
└── rendered/
└── staging/
└── services/
└── go-backend/
└── manifests.yaml # Generated by CI
```
Rendered manifests ensure what's in Git is exactly what deploys, PRs show actual Kubernetes changes (not just Helm values), and Git history provides a full audit trail.
## Key Design Decisions
| Decision | Rationale |
| -------------------------------------- | ---------------------------------------- |
| **Change detection with path filters** | Fast CI, minimal rebuilds in a monorepo |
| **Versioned image tags** | Immutable, traceable, rollback-friendly |
| **Rendered manifests in Git** | ArgoCD deploys exactly what's committed |
| **Staging auto-deploy on merge** | Merged code is always running in staging |
# Image CVE Scanning
Source: https://kubestarterkit.com/features/08-image-scanning
Automated vulnerability scanning for container images across environments
## Overview
The image CVE scanning workflow automatically scans container images for known vulnerabilities:
* **Scheduled scans:** Runs daily against production images to catch newly disclosed CVEs
* **On-demand scanning:** Can be triggered manually for any environment or specific images
* **Comprehensive coverage:** Extracts images from rendered Kubernetes manifests plus manually-specified images
* **Detailed reporting:** Generates SBOM and vulnerability reports with severity breakdowns
## What's Included
### Workflow Structure
```
.github/
├── workflows/
│ └── security-image-cve-scan.yml # Main scanning workflow
├── scripts/
│ └── extract-images.sh # Image extraction script
└── security/
└── additional-images.yaml # Extra images to scan
```
### Scanning Process
Collects container images from rendered Kubernetes manifests for the target environment, plus any additional images from configuration.
Creates a Software Bill of Materials (SBOM) for each image using [Syft](https://github.com/anchore/syft).
Analyzes each SBOM with [Grype](https://github.com/anchore/grype) to identify known CVEs.
Consolidates results into a summary report with vulnerability counts by severity.
### Image Discovery
Images are extracted from multiple sources:
1. **Kubernetes manifests:** Parses `image:` and `imageName:` fields from `kubernetes/rendered//`
2. **Additional images file:** Manual list in `.github/security/additional-images.yaml` for images not in manifests (e.g., operator-pulled images, sidecar injectors)
3. **Workflow input:** Ad-hoc images passed via workflow dispatch
```yaml theme={null}
# .github/security/additional-images.yaml
images:
# Example entries for images not in standard manifests:
# - docker.io/library/postgres:15-alpine
# - gcr.io/istio-release/proxyv2:1.20.0
```
### Vulnerability Scanning
Each image is scanned in a separate parallel job using GitHub Actions matrix strategy, enabling fast scans even with many images. The workflow uses the Anchore toolchain:
| Tool | Purpose |
| ----------------------------------------- | ---------------------------------------------- |
| [Syft](https://github.com/anchore/syft) | Generates CycloneDX SBOM from container images |
| [Grype](https://github.com/anchore/grype) | Matches SBOM packages against CVE databases |
The workflow handles both public images and private ECR images (via OIDC authentication).
### Report Output
The workflow generates a GitHub Actions job summary with:
* **Severity breakdown:** Critical, High, Medium, Low, Negligible, Unknown counts
* **Per-image results:** Table showing vulnerability counts for each scanned image
* **Scan artifacts:** Full Grype JSON output and SBOMs retained for 30 days
#### 🔍 CVE Scan Report
**Environment:** production\
**Scan Date:** YYYY-MM-DD HH:MM:SS UTC
#### Summary
| Severity | Count |
| ------------ | ----- |
| 🔴 Critical | 0 |
| 🟠 High | 0 |
| 🟡 Medium | 8 |
| 🔵 Low | 12 |
| ⚪ Negligible | 24 |
| ❓ Unknown | 0 |
**Successful scans:** 5\
**Failed scans:** 0
#### Results by Image
| Image | Status | Critical | High | Medium | Low |
| ---------------------------------------- | ------ | -------- | ---- | ------ | --- |
| `ghcr.io/cloudnative-pg/postgresql:17.2` | ✅ | 0 | 0 | 4 | 5 |
| `/go-backend:0.2.1` | ✅ | 0 | 0 | 2 | 3 |
| `/go-backend/migrations:0.2.1` | ✅ | 0 | 0 | 2 | 3 |
| `quay.io/argoproj/argocd:v2.13.3` | ✅ | 0 | 0 | 0 | 1 |
| `docker.io/traefik:v3.6.5` | ✅ | 0 | 0 | 0 | 0 |
## Usage
The workflow runs daily at 5 AM UTC (midnight EST) against production images. It can also be triggered manually via `workflow_dispatch` to scan a specific environment (`production`, `staging`, or `none`) or ad-hoc images.
## Key Design Decisions
| Decision | Rationale |
| ----------------------------------- | -------------------------------------------------- |
| **Daily scheduled scans** | Catch newly disclosed CVEs in existing images |
| **Extract from rendered manifests** | Scan exactly what's deployed, not source templates |
| **Parallel per-image jobs** | Fast scanning with isolated failures |
| **SBOM generation** | Enables offline analysis and compliance reporting |
| **Fail-fast disabled** | Complete scan of all images even if some fail |
## Additional Configurability
### Custom Severity Thresholds
To fail the workflow on critical vulnerabilities, modify the `report` job:
```yaml theme={null}
- name: Fail on critical vulnerabilities
if: steps.report.outputs.total_critical != '0'
run: |
echo "::error::Found ${{ steps.report.outputs.total_critical }} critical vulnerabilities"
exit 1
```
### Slack/Email Notifications
Add a notification step after the report job to alert on scan results.
### Vulnerability Suppression
Use Grype's [ignore rules](https://github.com/anchore/grype#specifying-matches-to-ignore) for false positives or accepted risks.
# Release Management
Source: https://kubestarterkit.com/features/09-release-management
Automated release PRs with release-please and controlled production deployments
## The Problem
Production deployments need more ceremony than staging, but that doesn't mean they should be painful:
* **Manual release processes:** Someone maintains a changelog, bumps versions, creates tags, and hopes they didn't miss anything.
* **Changelog drift:** The changelog is always out of date because updating it is tedious and easy to forget.
* **Version confusion:** What's actually in production? What commits are included in v1.2.3? Good luck piecing that together from Git history.
* **Risky deployments:** Without a clear release boundary, production deployments become "let's deploy main and hope for the best."
## How Kube Starter Kit Addresses This
I've integrated [release-please](https://github.com/googleapis/release-please) to automate release management with a simple, controlled workflow:
**Automatic release PRs:** As commits land on main, release-please maintains a PR that accumulates changes and auto-generates a changelog.
**Conventional commits drive versioning:** Commit messages determine version bumps. `fix:` bumps patch, `feat:` bumps minor, `feat!:` or `BREAKING CHANGE` bumps major.
**Merge to release:** When you're ready for production, merge the release PR. That's it.
**Production deployment on release:** Merging the release PR triggers the production build and deployment pipeline, rendering manifests for production and letting ArgoCD sync.
## What's Included
### Release-Please Configuration
```
.github/
├── workflows/
│ └── release-please.yaml
└── release-please-config.json
```
### How It Works
Developers merge PRs with conventional commit messages (`feat:`, `fix:`, `chore:`, etc.).
release-please automatically updates (or creates) a release PR that includes all pending changes with a generated changelog.
The release PR shows exactly what will be released: version bump, changelog entries, and all included commits.
Merging the release PR creates a Git tag and GitHub release.
The release triggers the production build workflow, which renders manifests and deploys to production via ArgoCD.
### Conventional Commits
The versioning is driven by commit message prefixes:
| Prefix | Version Bump | Example |
| ------------------------------ | --------------------- | ------------------------------------------- |
| `fix:` | Patch (1.0.0 → 1.0.1) | `fix: resolve null pointer in auth handler` |
| `feat:` | Minor (1.0.0 → 1.1.0) | `feat: add user profile endpoint` |
| `feat!:` or `BREAKING CHANGE:` | Major (1.0.0 → 2.0.0) | `feat!: redesign authentication API` |
| `chore:`, `docs:`, `ci:` | Patch (1.0.0 → 1.0.1) | `chore: update dependencies` |
Note: `chore:` commits bump the patch version to ensure pre-release image tags (e.g., `1.2.4-rc0001-g4b5d2e7`) always reflect the upcoming release version.
### Release PR Example
When commits accumulate, the release PR looks like:
```markdown theme={null}
## [1.2.0](https://github.com/org/repo/compare/v1.1.0...v1.2.0)
### Features
* add user profile endpoint (#45)
* support bulk operations in API (#42)
### Bug Fixes
* resolve null pointer in auth handler (#48)
* fix pagination on large datasets (#44)
```
This becomes both the PR description and the GitHub release notes.
### Production Deployment Flow
```
main branch
│
├── commit: feat: new feature
├── commit: fix: bug fix
│
▼
Release PR (auto-maintained)
│
│ [Merge when ready]
▼
Git Tag + GitHub Release
│
▼
Production Build Workflow
│
├── Build changed containers
├── Render production manifests
└── Commit to rendered/production/
│
▼
ArgoCD syncs production cluster
```
### Multi-Application Releases
In a monorepo with multiple applications, each application has its own version and receives separate Git tags (e.g., `services/go-backend@1.2.3`). However, release-please groups all pending releases into a single PR.
This gives you:
* **Independent versioning:** Each application's version reflects its own changes
* **Coordinated releases:** One PR to review and merge for all applications ready to release
* **Simpler workflow:** No need to manage multiple release PRs
The grouping behavior is configurable if you prefer separate release PRs per application.
## Key Design Decisions
| Decision | Rationale |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| **release-please over manual releases** | Automated changelog generation and version bumping eliminates manual toil and human error. |
| **Conventional commits** | Commit messages become meaningful. Version bumps are predictable and auditable. |
| **PR-based releases** | The release PR provides a clear review point before production. You can see exactly what's included. |
| **Separate staging and production pipelines** | Staging deploys on every merge for fast feedback. Production deploys only on release for controlled rollouts. |
| **Same rendered manifests pattern** | Production uses the same pattern as staging: rendered manifests in Git. Consistency across environments. |
## Workflow Summary
| Event | Staging | Production |
| ----------------- | -------------------------------- | -------------------------------------------- |
| PR merged to main | Builds and deploys automatically | No action |
| Release PR merged | No action | Builds and deploys automatically |
| Rollback needed | Revert commit, auto-redeploys | Revert release commit or deploy previous tag |
This separation gives you fast iteration on staging while maintaining controlled, auditable production releases.
# Demo Applications
Source: https://kubestarterkit.com/features/10-demo-applications
Fully functional example applications demonstrating end-to-end patterns
## The Problem
Infrastructure and deployment tooling is only half the story. Teams often struggle with:
* **No reference implementations:** You have a Kubernetes cluster, but how should applications actually be structured to use it?
* **Missing the full picture:** Tutorials show isolated pieces (a Dockerfile here, a Helm chart there) but not how everything connects.
* **Infrastructure gaps:** How do you provision a database for your app? How do you manage credentials? How do you connect from a pod?
* **Packaging questions:** What's the right way to structure Kubernetes manifests? How much should go in the Helm chart vs. external configuration?
## How Kube Starter Kit Addresses This
The kit includes fully functional demo applications that demonstrate the complete path from source code to running in production:
**Complete wiring, minimal code:** The applications themselves are simple, but include full configuration and external resource integration: database connections, secrets management, health checks, and Terraform-provisioned AWS resources.
**End-to-end patterns:** Each demo shows the full lifecycle: source code, Dockerfile, Helm chart, Terraform for cloud resources, and integration with the deployment pipeline.
**Copy and adapt:** The demos are designed to be starting points. Fork them, rename them, and build your actual services on top of proven patterns.
## What's Included
### Application Structure
```
applications/
└── go-backend/
├── src/ # Application source code
│ ├── main.go
│ ├── go.mod
│ └── ...
├── Dockerfile # Container build
└── README.md
kubernetes/src/services/
└── go-backend/
├── Chart.yaml # Helm chart definition
├── values.yaml # Default values
├── values-staging.yaml # Environment overrides
├── values-production.yaml
└── templates/
├── deployment.yaml
├── service.yaml
├── ingress.yaml
└── ...
terraform/modules/
└── app-resources/
└── go-backend/ # Cloud resources (S3, RDS, etc.)
```
### Go Backend
A production-ready Go service demonstrating:
* HTTP server with health check endpoints (`/healthz`, `/readyz`)
* Structured logging
* Graceful shutdown handling
* Configuration via environment variables
* Database connectivity patterns
* Multi-stage Dockerfile for minimal image size
* Non-root user for security
* Proper signal handling
* Build-time metadata (version, commit SHA)
* Deployment with resource limits and requests
* Liveness and readiness probes
* Service and Ingress configuration
* ConfigMaps and Secrets integration
* Horizontal Pod Autoscaler setup
* S3 bucket for object storage
* RDS PostgreSQL database (optional)
* IAM roles for pod-level AWS access via Pod Identity
* Secrets stored in AWS Secrets Manager
### Infrastructure Integration
Each demo application shows how to provision and connect to cloud resources:
```hcl theme={null}
# terraform/modules/app-resources/go-backend/main.tf
# S3 bucket for application data
module "s3_bucket" {
source = "terraform-aws-modules/s3-bucket/aws"
bucket = "${module.this.id}-data"
}
# IAM policy for S3 bucket access
resource "aws_iam_policy" "s3_access" {
name = "${module.this.id}-s3-access"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]
Resource = "${module.s3_bucket.s3_bucket_arn}/*"
}
]
})
}
# Pod Identity for AWS access
module "pod_identity" {
source = "terraform-aws-modules/eks-pod-identity/aws"
name = "${var.eks_cluster_name}-go-backend"
additional_policy_arns = {
S3Access = aws_iam_policy.s3_access.arn
}
associations = {
this = {
cluster_name = var.eks_cluster_name
namespace = var.kubernetes_namespace
service_account = var.kubernetes_service_account
}
}
}
```
Pod Identity associations are managed by Terraform. The application's ServiceAccount automatically receives AWS credentials without requiring annotations.
### Secrets Management
Demo applications use External Secrets to pull credentials from AWS Secrets Manager:
```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: go-backend-db
spec:
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: go-backend-db-credentials
data:
- secretKey: DATABASE_URL
remoteRef:
key: go-backend/database
property: url
```
No credentials in Git. No manual secret creation. Secrets sync automatically from AWS.
### Environment Configuration
Each application supports per-environment configuration:
```yaml theme={null}
# values.yaml (defaults)
replicaCount: 1
resources:
requests:
cpu: 100m
memory: 128Mi
```
```yaml theme={null}
# values.staging.yaml
replicaCount: 2
```
```yaml theme={null}
# values.production.yaml
replicaCount: 3
resources:
requests:
cpu: 500m
memory: 512Mi
```
The CI/CD pipeline renders the appropriate values for each environment.
## Using Demo Apps as Templates
Duplicate an existing demo app directory and rename it for your new service.
Replace the demo logic with your actual service implementation.
Adjust the cloud resources to match your application's needs (different database, additional S3 buckets, etc.).
Configure resource limits, replica counts, and environment variables for your service.
Add your new service to the change detection filters in the CI/CD workflow.
Commit and push. The pipeline builds, renders manifests, and ArgoCD deploys.
## Key Design Decisions
| Decision | Rationale |
| ---------------------------------- | -------------------------------------------------------------------------------------------------- |
| **Full applications, not stubs** | Teams learn better from working examples than from documentation. Real code shows real patterns. |
| **Terraform alongside Kubernetes** | Applications often need cloud resources. Showing both together demonstrates the complete workflow. |
| **Pod Identity for AWS access** | Pod-level IAM roles are more secure than shared credentials. The demos show the proper pattern. |
| **External Secrets integration** | Secrets management is often an afterthought. Including it in demos makes it the default pattern. |
| **Per-environment configuration** | Demonstrates how to handle environment differences with Kustomize, Helm, or Timoni. |
# Local Development
Source: https://kubestarterkit.com/features/11-local-development
KinD, Tilt, and mirrord for fast local Kubernetes development
## Overview
The kit includes a complete local development environment using three complementary tools:
* **[KinD](https://kind.sigs.k8s.io/) (Kubernetes in Docker):** A real multi-node Kubernetes cluster running locally
* **[Tilt](https://tilt.dev/):** Continuous development that rebuilds and redeploys on file changes
* **[mirrord](https://mirrord.dev/):** Run services directly on your host machine while connected to the KinD cluster
## How It Works
### KinD Cluster
KinD creates a real Kubernetes cluster using Docker containers as nodes. The kit's configuration provides:
**Multi-node cluster:** 1 control-plane + 2 worker nodes, matching a realistic production topology.
**Local container registry:** Images built locally are pushed to `localhost:5001` and pulled by the cluster, avoiding the need for external registries during development.
**Stable networking:** The cluster uses a dedicated Docker network (`172.20.0.0/16`) with [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind) to assign real LoadBalancer IPs. Ingress gets `172.20.0.100`, making URLs predictable.
**sslip.io for DNS:** Services are accessible via URLs like `go-backend.172-20-0-100.sslip.io` without `/etc/hosts` modifications.
### Tilt
Tilt watches your source code and Kubernetes manifests, automatically rebuilding containers and redeploying when files change.
**Hierarchical Tiltfiles:** The main `Tiltfile` includes infrastructure and services, each with their own `Tiltfile`. This mirrors the ArgoCD app-of-apps structure.
**Dependency ordering:** Resources declare dependencies (e.g., apps wait for cloudnative-pg operator), ensuring correct startup order.
**Smart rebuilds:** Only affected resources rebuild. Change application code and only that container rebuilds. Change a Helm value and only that release updates.
**Live UI:** Tilt's web UI shows resource status, logs, and build times. Click a resource to see its logs, trigger manual rebuilds, or debug failures.
### mirrord
While Tilt runs your services inside containers in the KinD cluster, mirrord lets you run a service directly on your host machine while staying connected to the cluster. This provides several advantages for debugging and profiling:
**Native debugging:** Attach your IDE's debugger directly to the process without container layers. Set breakpoints, step through code, and inspect variables with full IDE integration.
**Profiling tools:** Use host-native profilers (pprof, perf, Instruments) without containerization overhead or special configuration.
**Faster iteration:** Skip container builds entirely; just recompile and restart. Your local binary intercepts traffic that would go to the in-cluster pod.
**Steal traffic:** Redirect all traffic from a pod to your local process. Your local code handles real requests while you debug.
**Access cluster resources:** Your local process can reach cluster-internal services (databases, APIs) as if it were running in the cluster.
**Inherit environment:** Environment variables and secrets from the target pod are available locally.
## Development Workflows
### Local Kubernetes (Tilt)
Best for: Testing complete deployments including Kubernetes resources, databases, and ingress.
```
local/
├── kind/
│ └── cluster-config.yaml # Cluster definition
└── tilt/
└── Tiltfile # Main entry point
```
Tilt deploys:
* **Infrastructure:** cert-manager, cloudnative-pg, traefik
* **Applications:** Three variants of go-backend (Kustomize, Helm, Timoni)
Each application gets its own PostgreSQL database, runs migrations, and is accessible via ingress.
## Local Configuration
Each component has `values.local.yaml` or a `local/` overlay:
```
kubernetes/src/
├── infrastructure/
│ ├── traefik/values.local.yaml # LoadBalancer IP
│ └── cloudnative-pg/values.local.yaml
└── services/
├── go-backend/local/ # Kustomize overlay
├── go-backend-helm/values.local.yaml
└── go-backend-timoni/values.local.yaml
```
These configure:
* `storageClass: standard` for local PVCs (KinD's default)
* Image tags pointing to `localhost:5001` registry
* sslip.io hostnames for ingress
The same templating tools (Kustomize, Helm, Timoni) are used locally and in CI, ensuring consistency.
# SOC2 & Compliance Frameworks
Source: https://kubestarterkit.com/features/12-compliance-frameworks
How Kube Starter Kit features map to SOC2 controls and other compliance requirements
## Overview
Kube Starter Kit is designed with security and auditability as core principles. While the kit itself doesn't make you SOC2 compliant (that requires organizational policies, procedures, and a formal audit), it provides the technical foundation that addresses many SOC2 Trust Services Criteria.
This page maps Kube Starter Kit features to specific SOC2 controls and explains how they support your compliance posture.
## SOC2 Trust Services Criteria Coverage
SOC2 audits evaluate controls across five Trust Services Criteria. Kube Starter Kit primarily addresses **Security (Common Criteria)** and **Availability**, with supporting capabilities for **Processing Integrity** and **Confidentiality**.
### Security (Common Criteria)
The Security category forms the foundation of SOC2 and is required for all audits.
#### CC6.1 - Logical Access Security
> *The entity implements logical access security software, infrastructure, and architectures over protected information assets to protect them from security events.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ----------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Role-based access control** | [User Management](/features/04-user-management) | Single `users.yaml` defines AWS SSO users, groups, and permission sets. GitHub teams mirror AWS groups for consistent RBAC. |
| **Centralized identity management** | [User Management](/features/04-user-management) | AWS IAM Identity Center provides centralized authentication. No long-lived credentials; users authenticate via SSO portal. |
| **Least privilege enforcement** | [User Management](/features/04-user-management) | Pre-configured permission sets (Admin, PowerUser, ReadOnly) with AWS-managed policies. Easy to add custom permission sets for more granular access. |
| **Network segmentation** | [AWS Architecture](/features/03-aws-architecture) | EKS nodes run in private subnets with no public IPs. Workloads are isolated from direct internet access. |
| **Multi-account isolation** | [AWS Architecture](/features/03-aws-architecture) | Staging and production in separate AWS accounts. Hard boundary prevents cross-environment access or accidental impact. |
**Evidence artifacts:**
* Git history of `users.yaml` changes showing access reviews
* AWS IAM Identity Center user/group listings
* Terraform state showing VPC and subnet configurations
***
#### CC6.2 - Access Provisioning and Deprovisioning
> *Prior to issuing system credentials and granting system access, the entity registers and authorizes new internal and external users.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ------------------------------ | ----------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Documented access requests** | [User Management](/features/04-user-management) | All access changes are pull requests. PR description documents who, what, and why. |
| **Approval workflow** | [User Management](/features/04-user-management) | Branch protection requires PR approval before merge. Access changes get code review. |
| **Automated provisioning** | [Terraform + Terramate](/features/01-terraform) | Merging approved PRs triggers Terraform apply. No manual AWS console access needed. |
| **Timely deprovisioning** | [User Management](/features/04-user-management) | Remove user from `users.yaml`, open PR, merge. Access revoked across GitHub and AWS simultaneously. |
**Evidence artifacts:**
* Pull request history showing access change approvals
* Git blame on `users.yaml` showing who approved each change
* Terraform plan outputs showing exact access modifications
***
#### CC6.3 - Access Removal
> *The entity removes access to protected information assets when appropriate.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| -------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Single source of truth** | [User Management](/features/04-user-management) | One file controls all access. Removing a user entry revokes GitHub org membership, team access, and AWS SSO access. |
| **Audit trail** | [GitOps](/features/05-gitops) | Git history provides immutable record of when access was removed and by whom. |
| **No orphaned access** | [Terraform + Terramate](/features/01-terraform) | Terraform manages state declaratively. Users not in `users.yaml` are removed on apply. |
**Evidence artifacts:**
* Git commits showing user removal
* Terraform destroy plans for removed users
* AWS CloudTrail logs showing SSO user deletion
***
#### CC6.6 - Protection Against Threats Outside System Boundaries
> *The entity implements logical access security measures to protect against threats from sources outside its system boundaries.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ------------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **TLS encryption** | [Kubernetes Baseline](/features/06-k8s-baseline) | cert-manager automatically provisions and renews Let's Encrypt certificates for all ingress endpoints. |
| **No default credentials** | [Kubernetes Baseline](/features/06-k8s-baseline) | External Secrets pulls credentials from AWS Secrets Manager. No secrets in Git, no default passwords. |
| **Network perimeter controls** | [AWS Architecture](/features/03-aws-architecture) | NLB with security groups controls inbound traffic. Private subnets prevent direct access to nodes. |
| **Container vulnerability scanning** | [Image Scanning](/features/08-image-scanning) | Daily CVE scans against production images. SBOM generation for supply chain visibility. |
**Evidence artifacts:**
* cert-manager Certificate resources showing valid TLS
* CVE scan reports from GitHub Actions
* VPC security group configurations
***
#### CC6.7 - Transmission Encryption
> *The entity restricts the transmission, movement, and removal of information to authorized internal and external users and processes.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ------------------------------------ | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Encryption in transit** | [Kubernetes Baseline](/features/06-k8s-baseline) | All external traffic terminates TLS at traefik. Internal cluster traffic uses Kubernetes network policies. |
| **Automated certificate management** | [Kubernetes Baseline](/features/06-k8s-baseline) | cert-manager handles certificate lifecycle. No manual certificate handling or risk of expiration. |
**Evidence artifacts:**
* Ingress resources with TLS configuration
* Certificate resources showing renewal history
***
#### CC6.8 - Malicious Software Prevention
> *The entity implements controls to prevent or detect and act on the introduction of unauthorized or malicious software.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ---------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **Container image scanning** | [Image Scanning](/features/08-image-scanning) | Grype scans all container images for known CVEs. Reports generated with severity breakdowns. |
| **SBOM generation** | [Image Scanning](/features/08-image-scanning) | Syft generates Software Bill of Materials for each image. Enables tracking of vulnerable dependencies. |
| **Immutable deployments** | [CI/CD Pipeline](/features/07-ci-cd-pipeline) | Container images are tagged with version and commit hash. Tags are never overwritten. |
**Evidence artifacts:**
* CVE scan reports and SBOM artifacts
* ECR image tags showing immutable versioning
* GitHub Actions workflow runs
***
### CC8.1 - Change Management
> *The entity authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ------------------------------ | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **Change authorization** | [GitOps](/features/05-gitops) | All changes require pull request approval. Branch protection enforces review requirements. |
| **Change documentation** | [GitOps](/features/05-gitops) | Git commits document what changed. PR descriptions explain why. Rendered manifests show exact Kubernetes changes. |
| **Separation of environments** | [AWS Architecture](/features/03-aws-architecture) | Changes deploy to staging first. Production deployment requires separate approval/promotion. |
| **Automated testing** | [CI/CD Pipeline](/features/07-ci-cd-pipeline) | CI runs on every PR. Manifest validation ensures rendered output matches committed files. |
| **Rollback capability** | [GitOps](/features/05-gitops) | Reverting a Git commit triggers ArgoCD rollback. Previous state is always recoverable. |
**Evidence artifacts:**
* Pull request history with approvals
* Git diff showing exact changes per deployment
* ArgoCD sync history showing deployments
***
#### CC8.2 - Infrastructure and Software Changes
> *The entity authorizes, documents, and implements changes to infrastructure and software.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| -------------------------- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Infrastructure as Code** | [Terraform + Terramate](/features/01-terraform) | All AWS infrastructure defined in Terraform. Changes visible in PR diffs. |
| **Change visibility** | [Terramate](/features/02-terramate) | `terraform plan` output posted to PRs. Reviewers see exactly what will change before approval. |
| **Drift detection** | [Terramate](/features/02-terramate) | Scheduled workflows detect infrastructure drift. Alerts when actual state differs from code. |
| **Kubernetes changes** | [GitOps](/features/05-gitops) | Rendered manifests committed to Git. ArgoCD shows diff between desired and actual state. |
**Evidence artifacts:**
* Terraform plan outputs in PRs
* Drift detection workflow results
* ArgoCD application sync status
***
### A1.1, A1.2 - Availability
> *The entity maintains, monitors, and evaluates current processing capacity and use of system components to support availability.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ----------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| **Capacity management** | [Kubernetes Baseline](/features/06-k8s-baseline) | Karpenter auto-provisions nodes based on workload demand. Removes underutilized capacity automatically. |
| **High availability** | [AWS Architecture](/features/03-aws-architecture) | EKS spans 3 availability zones. Node failures don't affect application availability. |
| **Observability** | [Kubernetes Baseline](/features/06-k8s-baseline) | SigNoz provides metrics, logs, and traces. Enables monitoring of system health and performance. |
| **Database resilience** | [Kubernetes Baseline](/features/06-k8s-baseline) | CloudNativePG provides automated failover for PostgreSQL. Backups to S3 enable point-in-time recovery. |
**Evidence artifacts:**
* Karpenter provisioner logs
* SigNoz dashboards and alerts
* CloudNativePG backup records
***
### C1.1, C1.2 - Confidentiality
> *The entity identifies and maintains confidential information to meet the entity's objectives related to confidentiality.*
| Control Requirement | Kube Starter Kit Feature | Implementation |
| ---------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Secrets management** | [Kubernetes Baseline](/features/06-k8s-baseline) | External Secrets syncs credentials from AWS Secrets Manager. Secrets never stored in Git. |
| **Encryption at rest** | [AWS Architecture](/features/03-aws-architecture) | EBS volumes, S3 buckets, and Secrets Manager all encrypted with KMS by default. |
| **Access to secrets** | [Kubernetes Baseline](/features/06-k8s-baseline) | Pod Identity provides scoped AWS credentials. Pods only access secrets they're authorized for. |
**Evidence artifacts:**
* ExternalSecret resources (not containing actual secret values)
* AWS Secrets Manager access logs
* Pod Identity association configurations
***
## Control Matrix Summary
| SOC2 Control | Primary Feature | Evidence Location |
| ------------------------------- | ------------------- | ------------------------------ |
| CC6.1 - Logical Access | User Management | `users.yaml` + Git history |
| CC6.2 - Access Provisioning | User Management | Pull requests |
| CC6.3 - Access Removal | User Management | Git commits + Terraform state |
| CC6.6 - External Threats | Image Scanning, TLS | CVE reports, Certificates |
| CC6.7 - Transmission Encryption | cert-manager | Ingress TLS configs |
| CC6.8 - Malicious Software | Image Scanning | SBOM + CVE reports |
| CC8.1 - Change Management | GitOps | Git history + ArgoCD |
| CC8.2 - Infrastructure Changes | Terraform/Terramate | Plan outputs + drift detection |
| A1.1/A1.2 - Availability | Karpenter, Multi-AZ | Provisioner logs, SigNoz |
| C1.1/C1.2 - Confidentiality | External Secrets | Secrets Manager audit logs |
***
## Other Compliance Frameworks
The same technical controls that support SOC2 also map to other compliance frameworks:
### HIPAA (Healthcare)
| HIPAA Requirement | Kube Starter Kit Feature |
| -------------------------------------- | ------------------------------- |
| Access Controls (§164.312(a)(1)) | User Management with RBAC |
| Audit Controls (§164.312(b)) | GitOps audit trail + CloudTrail |
| Transmission Security (§164.312(e)(1)) | TLS via cert-manager |
| Integrity Controls (§164.312(c)(1)) | Immutable container images |
### PCI-DSS (Payment Card)
| PCI-DSS Requirement | Kube Starter Kit Feature |
| ---------------------------- | --------------------------------- |
| Req 1: Network Segmentation | Private subnets, security groups |
| Req 2: Secure Configurations | Infrastructure as Code |
| Req 6: Secure Development | CI/CD with vulnerability scanning |
| Req 7: Restrict Access | RBAC via User Management |
| Req 8: Identify Users | AWS IAM Identity Center |
| Req 10: Track Access | Git audit trail + CloudTrail |
### ISO 27001
| ISO 27001 Control | Kube Starter Kit Feature |
| ------------------------ | ------------------------ |
| A.9 Access Control | User Management |
| A.12 Operations Security | GitOps, CI/CD |
| A.14 System Development | IaC, rendered manifests |
| A.16 Incident Management | Observability (SigNoz) |
***
## Preparing for Audit
While Kube Starter Kit provides the technical controls, a SOC2 audit also requires:
1. **Policies and Procedures:** Written documentation of your access control policy, change management process, incident response plan, etc.
2. **Evidence Collection:** Regular exports of:
* Git history for access changes
* Terraform plan outputs
* CVE scan reports
* ArgoCD sync history
3. **Access Reviews:** Periodic review of `users.yaml` to verify access is still appropriate.
4. **Risk Assessments:** Documentation of how you triage and remediate CVE findings.
5. **Training Records:** Evidence that team members understand security procedures.
**Start early:** Begin collecting evidence before your audit period. The Git-based workflow automatically creates an audit trail, but you'll want to establish the rhythm of periodic reviews and documentation.
# Kube Starter Kit
Source: https://kubestarterkit.com/index
From zero to production Kubernetes in days, not months.
# The Kubernetes platform you'd build if you had the time
Getting Kubernetes right is hard. Not the “spin up a cluster” part, that’s easy. The hard part is everything else: infrastructure as code and automation, implementing GitOps and deployments, managing secrets securely, establishing observability, meeting compliance requirements, and making it all work across multiple environments without spending months on it.
I've built and maintained Kubernetes platforms at multiple companies. Each time, I found myself solving the same problems, making the same architectural decisions, and wishing I had a solid starting point instead of building from scratch.
**Kube Starter Kit is that starting point.**
It's an opinionated, production-ready platform for **AWS and EKS** that gives you everything you need to run applications on Kubernetes: infrastructure, deployment pipelines, and baseline components, all wired together and ready to go.
## What's Included
Modular, well-structured Terraform for AWS with multi-environment support via Terramate
Stack-based Terraform orchestration with Terramate; change detection, outputs sharing, and CI/CD
Multi-account setup with VPC, EKS, and secure account boundaries
Terraform-managed GitHub and AWS IAM Identity Center users and permissions
ArgoCD-based GitOps for declarative, auditable deployments
Curated set of components: ingress, cert-manager, external-dns, secrets, observability, and more
Automated container builds and staging deployments on every merge to main
Automated vulnerability scanning for container images with daily scheduled scans
Automated release PRs with release-please and controlled production deployments
Fully functional example applications demonstrating end-to-end patterns
KinD, Tilt, and mirrord for fast local Kubernetes development
How Kube Starter Kit features map to SOC2 controls and other compliance requirements
## Who This Is For
Kube Starter Kit is built for **engineering teams at early-stage companies** who:
* Are confident Kubernetes is the right choice for their infrastructure
* Want to get to production quickly without cutting corners
* Value having an opinionated path from someone who's done this before
* Would rather customize a working system than build one from scratch
## What to Expect
**You own the infrastructure.** This is your platform running in your AWS account. No managed service dependency, no third-party with access to your environment. Full control, full responsibility.
**Standard tools, no abstraction layer.** Unlike OpenShift or Tanzu, there's no proprietary UI or vendor lock-in. It's standard Kubernetes with standard tools. Your team's existing knowledge transfers directly.
**A starting point you make your own.** Fork the repo and customize it to fit your needs. All future updates to the kit are available to you, but you're also free to diverge. It's a foundation, not a constraint.
## Get Started
Learn about each component and the decisions behind them
Get access to the kit and optional consulting support
# Licensing / Pricing
Source: https://kubestarterkit.com/license-pricing
Source available, not open source
## License
Kube Starter Kit is **source-available**, not open source.
* **Learning and reference?** Go for it. Read the code, understand the patterns, use it for personal projects or experimentation.
* **Using it to run your business?** Please purchase a license.
See the full [LICENSE](https://github.com/DevOps-Directive/kube-starter-kit/blob/main/LICENSE) for details. When you purchase a license, you get perpetual rights: you own the code and can use it indefinitely.
## Pricing
### Unpaid License
Free for learning, reference, personal projects, and experimentation.
### Paid Licenses
**\$8,500** one-time
* Use of all Terraform modules, Kubernetes manifests, and CI/CD pipelines
* Access to all future updates
* **Async support** (Slack/Teams/Discord) for 6 months
Best for teams with strong Kubernetes and AWS experience who want a solid starting point.
**\$18,500** one-time
* Everything in Self-Serve
* **60 hours of consulting** to help you roll out, operationalize, and extend
* Pair on initial setup, customization, and deployment
* Priority access to answer questions and review your implementation
Best for teams who want hands-on guidance during setup.
### Additional Consulting Hours
Need more help after your initial hours? Additional consulting is available at **\$235/hr**.
Use extra hours for:
* Extending the platform with custom features
* Training your team on operations and maintenance
* Troubleshooting and debugging
* Architecture reviews as your needs evolve
***
## Questions?
Not sure if you need a license? Just ask: [sid@devopsdirective.com](mailto:sid@devopsdirective.com)
I'm reasonable. If you're a tiny bootstrapped startup or have an unusual situation, reach out and we'll figure it out.
# Overview
Source: https://kubestarterkit.com/usage/getting-started/00-overview
Overview of the deployment process for Kube Starter Kit
This page provides a high-level overview of the deployment process. Each section links to detailed documentation with step-by-step instructions.
## [1. Prerequisites](/usage/getting-started/01-prerequisites)
Covers the AWS account structure, GitHub organization requirements, domain setup, and tooling you'll need before starting.
## [2. Repository Setup](/usage/getting-started/02-repository-setup)
Fork the repository and install the development tools.
## [3. Bootstrap AWS Accounts](/usage/getting-started/03-bootstrap-accounts)
Create the foundational resources that Terraform needs to manage everything else: state storage, IAM roles, and cross-account trust relationships. This is a one-time setup that enables both local Terraform runs and CI/CD automation.
## [4. Configure Access](/usage/getting-started/04-configure-access)
Set up user access for AWS and GitHub from a single configuration file.
## [5. Configure CI/CD Integrations](/usage/getting-started/05-configure-integrations)
Configure GitHub Actions to authenticate with AWS and set up any optional integrations like Terramate Cloud.
## [6. Deploy Infrastructure](/usage/getting-started/06-deploy-infrastructure)
Deploy the core AWS infrastructure: networking, EKS cluster, and application-specific resources.
This step is performed for each deployment environment (e.g. staging and production).
## [7. Cluster Access](/usage/getting-started/07-cluster-access)
Configure kubectl access to your EKS cluster through the bastion host.
This step is performed for each deployment environment (e.g. staging and production).
## [8. Deploy Kubernetes Baseline](/usage/getting-started/08-deploy-kubernetes-baseline)
Bootstrap ArgoCD and the infrastructure components it depends on. Once running, ArgoCD takes over and manages all remaining cluster resources.
This step is performed for each deployment environment (e.g. staging and production).
## [9. Local Development](/usage/getting-started/09-local-development-setup)
Set up a local Kubernetes environment for development and testing.
# Prerequisites
Source: https://kubestarterkit.com/usage/getting-started/01-prerequisites
What you need before getting started
## Required
### AWS Account
You need an AWS account where the infrastructure will be deployed. The kit uses [AWS Organizations](https://aws.amazon.com/organizations/) to manage multiple accounts (infrastructure, staging, production).
If your AWS account is brand new, you may need to request a quota increase for AWS Organizations. The default limit is 5 accounts, which may not be sufficient. Request an increase through the [Service Quotas console](https://console.aws.amazon.com/servicequotas/).
#### Setting Up Multiple Accounts with Control Tower
You can set up your AWS accounts however you prefer. The kit just needs an management account, infrastructure account, ECR account, staging account, and production account to exist. If you already have a multi-account structure, you can skip this section.
If you're starting fresh, [AWS Control Tower](https://aws.amazon.com/controltower/) is a good option. It provides:
* **Account Factory:** Provision new accounts with consistent baseline configurations
* **Guardrails:** Pre-configured governance rules (SCPs) for security and compliance
* **IAM Identity Center (SSO):** Centralized access management across all accounts
* **Landing Zone:** A well-architected multi-account environment out of the box
To set up Control Tower:
1. Enable Control Tower in your **management** account
2. Create an **Infrastructure** account (the entrypoint and hub for IaC automations)
3. Create an **ECR** account (for the Elastic Container Registry used by all environments)
4. Create a **Staging** account (for the staging EKS cluster and application resources)
5. Create a **Production** account (for the production EKS cluster and application resources)
Control Tower automatically sets up AWS IAM Identity Center, which the kit uses for authenticating to each account via SSO.
#### Configure IAM Identity Center for Initial Access
Before you can run Terraform to bootstrap the accounts, you need at least one admin user in IAM Identity Center with access to the Infrastructure account. This is a one-time manual setup.
If you used Control Tower, IAM Identity Center is already enabled. Otherwise, enable it in your management account:
1. Navigate to [IAM Identity Center](https://console.aws.amazon.com/singlesignon/) in the AWS Console
2. Click **Enable** and choose your identity source (use the built-in Identity Center directory for simplicity)
1. Go to **Users** → **Add user**
2. Enter your email address and name
3. Complete the email verification process
1. Go to **Groups** → **Create group**
2. Name it `Admin`
3. Add your user to the group
1. Go to **Permission sets** → **Create permission set**
2. Choose **Predefined permission set** → **AdministratorAccess**
3. Set session duration (12 hours recommended for development)
4. Create the permission set
1. Go to **AWS accounts**
2. Select the **Infrastructure** account
3. Click **Assign users or groups**
4. Select the **Admin** group and the **AdministratorAccess** permission set
You only need to do this for the Infrastructure account. Terraform will manage assignments to other accounts after bootstrapping.
Go to **Settings** and note your **AWS access portal URL** (e.g., `https://d-xxxxxxxxxx.awsapps.com/start`). You'll need this to configure Leapp.
After bootstrapping, the kit manages IAM Identity Center users, groups, and assignments via Terraform in the `user-management` stack. This initial manual setup just gets you access to run Terraform for the first time.
For more details, see the [AWS IAM Identity Center documentation](https://docs.aws.amazon.com/singlesignon/latest/userguide/getting-started.html).
### GitHub Organization
A GitHub organization where the repository will live. The CI/CD pipelines use GitHub Actions with OIDC authentication to AWS, and the Terraform GitHub provider manages org-level resources like team memberships.
You need **Owner** permissions on the GitHub organization to complete the setup. This is required to:
* Create and configure repositories
* Install GitHub Apps (octo-sts for token management)
* Manage organization members and teams
* Configure repository secrets and OIDC settings
### Domain Name
A domain name is required for ingress routing to your services (e.g., `api.example.com`). The kit creates Route53 hosted zones and uses external-dns to automatically manage DNS records.
You can either:
* **Register a new domain** through AWS Route53 or any registrar
* **Use a subdomain** of an existing domain by pointing nameservers to the Route53 hosted zone
## Optional
### SigNoz Cloud Account
[SigNoz](https://signoz.io/) provides observability (traces, metrics, logs). The kit includes pre-configured OpenTelemetry collectors that ship data to SigNoz Cloud.
If you don't set up SigNoz, the observability components simply won't send data anywhere. The rest of the platform works fine without it.
**NOTE:** other observability providers can also be swapped in instead of Signoz.
### Terramate Cloud Account
[Terramate Cloud](https://terramate.io/) enhances the Terraform workflow with:
* Drift detection dashboards
* PR preview comments showing planned changes
* Stack health monitoring
The kit works without Terramate Cloud; you just won't get the cloud features. The CLI orchestration still functions locally and in CI.
# Repository Setup
Source: https://kubestarterkit.com/usage/getting-started/02-repository-setup
Fork and customize the repository for your organization
## Overview
Kube Starter Kit is designed to be forked and customized for your organization. This page walks you through the initial repository setup: forking, cloning, and installing development tools.
## Fork and Clone
Fork the repository to your GitHub organization:
1. Navigate to the [Kube Starter Kit repository](https://github.com/DevOps-Directive/kube-starter-kit)
2. Click **Fork** and select your organization as the owner
3. Keep the repository name or rename it to match your conventions
```bash theme={null}
git clone "https://github.com//.git"
cd ""
```
The repository uses [mise](https://mise.jdx.dev/) to manage tool versions. Install it first:
```bash theme={null}
curl https://mise.run | sh
```
Then activate mise in your shell (add to your `.bashrc` or `.zshrc` for persistence):
```bash theme={null}
eval "$(~/.local/bin/mise activate bash)" # or zsh/fish
```
See the [mise installation docs](https://mise.jdx.dev/getting-started.html) for alternative installation methods.
With mise installed, install the project's tool dependencies:
```bash theme={null}
mise install
```
This installs Terramate, Terraform, kubectl, Helm, and other tools at the versions specified in `mise.toml`.
## Next Steps
You're now ready to proceed to [Bootstrap Accounts](/usage/getting-started/03-bootstrap-accounts) to set up AWS IAM roles and import GitHub organization members.
# Bootstrap Accounts
Source: https://kubestarterkit.com/usage/getting-started/03-bootstrap-accounts
Set up AWS IAM roles and import GitHub organization members
## Overview
Before deploying infrastructure via CI/CD, you need to bootstrap your AWS accounts and GitHub organization for Terraform management. This is a one-time setup that creates the foundation for all automation.
### Decisions
This setup step requires deciding the following:
#### Namespace
The **namespace** is a short prefix (3-5 characters) used to generate unique names for all AWS resources: S3 buckets, IAM roles, EKS clusters, etc. Choose something that identifies your organization.
| Examples | Description |
| -------- | ------------------------- |
| `acme` | Company name abbreviation |
| `myco` | Short identifier |
| `xyz` | Project code |
The default is `ksk` ("Kube Starter Kit"). You'll use this namespace consistently across all configuration, it cannot be easily changed later.
This "namespace" refers to the naming convention from [Cloud Posse's terraform-null-label](https://github.com/cloudposse/terraform-null-label), **not** a Kubernetes namespace.
#### Primary AWS Region
Choose a primary AWS region for your infrastructure. This region will host:
* The Terraform state S3 bucket
* Your EKS clusters (staging and production)
* Most other AWS resources
The default is `us-east-2`. Consider factors like latency to your users, service availability, and pricing when choosing.
You can deploy to multiple regions later, but the state bucket region cannot be changed without migrating state.
### How Cross-Account Access Works
The **Infrastructure account** is the central hub for all Terraform operations. There are two access paths:
1. **CI/CD (GitHub Actions)**: Authenticates via OIDC, then assumes roles in target accounts
2. **Human admins (IAM Identity Center)**: Authenticates via SSO to the Infrastructure account, then assumes roles in target accounts
```
GitHub Actions Human Admin (You)
│ │
│ OIDC │ IAM Identity Center (SSO)
▼ ▼
┌─────────────────────────────────────────────────────┐
│ Infrastructure Account │
│ │
│ ┌─────────────────┐ ┌────────────────────┐ │
│ │ GitHub OIDC │ │ SSO Admin Role │ │
│ │ Role │ │ (via Leapp) │ │
│ └─────────────────┘ └────────────────────┘ │
│ \ / │
│ ▼ ▼ │
│ ┌──────────────────────┐ │
│ │ Terraform State │ │
│ │ Bucket (S3) │ │
│ └──────────────────────┘ │
└─────────────────────────────────────────────────────┘
│ │ │ \
│ sts:AssumeRole (cross-account) │ \
▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Management │ │ ECR │ │ Staging │ │ Production │
│ Account │ │ Account │ │ Account │ │ Account │
│ ------------ │ │ ------------ │ │ ------------ │ │ ------------ │
│ IAM Role │ │ IAM Role │ │ IAM Role │ │ IAM Role │
│ (trusts │ │ (trusts │ │ (trusts │ │ (trusts │
│ infra) │ │ infra) │ │ infra) │ │ infra) │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
```
**For CI/CD (GitHub Actions):**
1. GitHub Actions authenticates via OIDC to the GitHub OIDC role in the Infrastructure account
2. That role assumes target account roles via cross-account IAM trust policies
3. Terraform runs with credentials for the target account, state stored in Infrastructure account
**For Human Admins:**
1. Admin authenticates to the Infrastructure account via IAM Identity Center (using Leapp)
2. The SSO role in Infrastructure account can assume target account roles
3. Admin runs Terraform locally with the same cross-account access as CI/CD
This pattern keeps credentials management simple while maintaining proper account isolation.
## What Gets Created
### Infrastructure Account (Central Hub)
| Resource | Purpose |
| -------------------- | ------------------------------------------------------------------------- |
| S3 bucket | Stores Terraform state for **all accounts** with versioning enabled |
| GitHub OIDC provider | Enables keyless authentication from GitHub Actions |
| GitHub OIDC IAM role | Role that GitHub Actions assumes; can then assume roles in other accounts |
### Each Target Account (Management, ECR, Staging, Production)
| Resource | Purpose |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Terraform IAM role | Admin role with trust policy allowing **both** the GitHub OIDC role (for CI/CD) **and** SSO admin role (for human admins) from the Infrastructure account to assume it |
| Route53 hosted zone | DNS zone for the environment (e.g., `staging.example.com`); not created in Management or ECR accounts |
## Prerequisites
### Required Accounts
Based on the [account structure](/features/03-aws-architecture#account-structure), you need these AWS accounts:
| Account | Purpose |
| ------------------ | ----------------------------------------------- |
| **Management** | AWS Organizations, IAM Identity Center |
| **Infrastructure** | Terraform state, GitHub OIDC, CI/CD automation |
| **ECR** | Container registry (shared across environments) |
| **Staging** | Staging environment resources |
| **Production** | Production environment resources |
## Bootstrapping Overview
Bootstrapping solves a chicken-and-egg problem: Terraform needs IAM roles and an S3 bucket to run, but we want Terraform to manage those resources. The solution is to manually create minimal resources, then let Terraform import and manage them.
Each account needs a bootstrap IAM role. The Infrastructure account additionally needs an S3 bucket for Terraform state.
The bootstrap scripts in `terraform/bootstrap/` must be run manually with AWS CLI credentials before Terraform can take over.
## Configure Leapp CLI
Before you can authenticate to AWS accounts, configure the Leapp CLI with your IAM Identity Center portal:
```bash theme={null}
leapp integration create \
--integrationType AWS-SSO \
--integrationAlias "My Organization" \
--integrationPortalUrl https://d-xxxxxxxxxx.awsapps.com/start \
--integrationRegion
```
Replace the portal URL with your IAM Identity Center URL (found in the AWS IAM Identity Center console under **Settings > Identity source**) and `` with your [primary region](#primary-aws-region).
For a GUI experience, [Leapp](https://www.leapp.cloud/) provides a desktop app for managing AWS SSO sessions. It discovers available accounts and permission sets automatically from your configured integration.
## Bootstrap the Infrastructure Account
The Infrastructure account is special, it hosts the S3 state bucket that all other accounts depend on.
Start a Leapp session for the Infrastructure account:
```bash theme={null}
leapp session start "Infrastructure"
aws sts get-caller-identity
```
You'll need this ARN for cross-account trust policies:
```bash theme={null}
mise run //terraform/bootstrap:get-sso-role-arn
```
Save this ARN, it looks like:
```
arn:aws:iam::INFRA_ACCOUNT_ID:role/aws-reserved/sso.amazonaws.com/REGION/AWSReservedSSO_AdministratorAccess_XXXXX
```
```bash theme={null}
mise run //terraform/bootstrap:create-state-bucket \
--bucket-name -gbl-infra-bootstrap-state \
--aws-region
```
S3 bucket names are globally unique. Replace `` with your [namespace](#namespace) and `` with your [primary region](#primary-aws-region).
Edit `terraform/config.tm.hcl` with your namespace and account details:
```hcl theme={null}
globals {
namespace = "" # Your chosen namespace
# Your SSO admin role in Infrastructure account (for local Terraform runs)
sso_admin_assume_role_arn = "arn:aws:iam:::role/aws-reserved/sso.amazonaws.com//AWSReservedSSO_AdministratorAccess_XXXXX"
# GitHub OIDC role (will be created by Terraform)
github_oidc_assume_role_arn = "arn:aws:iam:::role/-gbl-infra-bootstrap-github-oidc"
# S3 backend configuration (must match the bucket you created above)
backend_bucket = "-gbl-infra-bootstrap-state"
backend_region = ""
}
```
Replace the placeholders with your values from [Decisions](#decisions).
Propagate the configuration changes to generated files:
```bash theme={null}
cd terraform
terramate generate
```
This imports the S3 bucket and creates the GitHub OIDC provider and role:
```bash theme={null}
cd terraform
terramate run --tags infra:bootstrapping --parallel 1 -- terraform init
terramate run --tags infra:bootstrapping -- terraform apply
```
## Bootstrap Target Accounts
Each target account (Management, ECR, Staging, Production) needs an IAM role that can be assumed from the Infrastructure account.
### Manual Role Creation
For each account:
```bash theme={null}
leapp session start "Staging" # or Management, ECR, Production
aws sts get-caller-identity
```
```bash theme={null}
cd terraform/bootstrap
mise run create-terraform-iam-role-in-target-account \
--role-name \
--infra-sso-role-arn ""
```
Replace `` with the SSO role ARN you recorded during [Bootstrap the Infrastructure Account](#bootstrap-the-infrastructure-account).
Use the appropriate role name for each account:
| Account | Role Name |
| ---------- | ----------------------------------------- |
| Management | `-gbl-mgmt-bootstrap-admin` |
| ECR | `-gbl-ecr-bootstrap-admin` |
| Staging | `-gbl-staging-bootstrap-admin` |
| Production | `-gbl-prod-bootstrap-admin` |
This creates an IAM role with:
* `AdministratorAccess` policy attached
* Trust policy allowing your SSO role from the Infrastructure account to assume it
Repeat these steps for each target account.
### Terraform Takeover
Once the manual roles exist, Terraform can import and manage them. Terramate generates an `import` block in each root module to bring the manually-created role into Terraform state.
#### How Import Works
The Terramate template in `terraform/imports/mixins/modules/bootstrapping.tm.hcl` generates both the import block and module call for each bootstrapping stack:
```hcl theme={null}
# Generated in each root module (e.g., terraform/live/staging/global/bootstrapping/_main.tf)
import {
to = module.bootstrapping.module.iam_role.aws_iam_role.this[0]
id = "ksk-gbl-staging-bootstrap-admin" # Uses globals: namespace-environment-stage
}
module "bootstrapping" {
source = "../../../../modules/account-bootstrapping"
# ...
}
```
The import block ID is constructed from your configured globals (`namespace`, `environment`, `stage`) to match the role name you created manually (e.g., `"-gbl-staging-bootstrap-admin"`).
Import blocks must be in root modules, not child modules. This is a Terraform requirement. The `account-bootstrapping` module itself does not contain the import block; it's generated by Terramate at the root module level.
When you run `terraform apply`, Terraform:
1. Imports the existing IAM role into state (instead of trying to create it)
2. Updates the role's trust policy to allow **both** your SSO role **and** the GitHub OIDC role to assume it
This enables CI/CD pipelines to manage infrastructure going forward.
#### Apply the Bootstrapping Stacks
```bash theme={null}
cd terraform
# Log back into Infrastructure account
leapp session start "Infrastructure"
# Apply each account's bootstrapping stack
terramate run --tags management:bootstrapping -- terraform init
terramate run --tags management:bootstrapping -- terraform apply
terramate run --tags ecr:bootstrapping -- terraform init
terramate run --tags ecr:bootstrapping -- terraform apply
terramate run --tags staging:bootstrapping -- terraform init
terramate run --tags staging:bootstrapping -- terraform apply
terramate run --tags prod:bootstrapping -- terraform init
terramate run --tags prod:bootstrapping -- terraform apply
```
The `import` block was added in Terraform 1.5. It allows declarative imports without running `terraform import` commands manually. See the [Terraform import block documentation](https://developer.hashicorp.com/terraform/language/block/import) for more details.
## Configure Domain Nameservers
The bootstrapping stacks create Route53 hosted zones for Staging and Production (e.g., `staging.example.com`, `prod.example.com`). For DNS to work, you must configure your domain registrar to use the Route53 nameservers.
After applying the bootstrapping stacks, retrieve the nameservers for each hosted zone:
```bash theme={null}
# Get staging nameservers
terramate run --tags staging:bootstrapping -- terraform output hosted_zone_nameservers
# Get production nameservers
terramate run --tags prod:bootstrapping -- terraform output hosted_zone_nameservers
```
You'll see 4 nameservers like:
```
ns-123.awsdns-45.com
ns-678.awsdns-90.net
ns-111.awsdns-22.org
ns-333.awsdns-44.co.uk
```
How you configure nameservers depends on your setup:
**If using a subdomain** (e.g., `staging.example.com`):
* Add NS records in your parent domain's DNS pointing to the Route53 nameservers
* Example: Add NS records for `staging` subdomain pointing to the 4 nameservers above
**If using a dedicated domain** (e.g., `example-staging.com`):
* Update the domain's nameservers at your registrar (Namecheap, GoDaddy, Route53, etc.)
* Replace the default nameservers with the 4 Route53 nameservers
DNS changes can take up to 48 hours to propagate, but usually complete within minutes. Verify with:
```bash theme={null}
dig NS staging.example.com +short
```
You should see the Route53 nameservers in the response.
If you skip this step, external-dns and cert-manager will not work. DNS records created in Route53 won't resolve, and Let's Encrypt DNS-01 challenges will fail.
## Troubleshooting
### "Bucket already exists" error
S3 bucket names are globally unique. If the bucket name is taken:
1. Choose a different name with your organization prefix
2. Update `backend_bucket` in `terraform/config.tm.hcl`
3. Update any hardcoded references in bootstrapping stacks
### State file in wrong location
All Terraform state is stored in the Infrastructure account's S3 bucket, regardless of which account the resources are in. If you see state errors:
1. Verify `backend_bucket` and `backend_region` in `terraform/config.tm.hcl`
2. Ensure the GitHub OIDC role has S3 permissions in the Infrastructure account
3. Check that the bucket exists and has the expected state files
## Next Steps
With accounts bootstrapped, proceed to [Configure Access](/usage/getting-started/04-configure-access) to set up CI/CD and user access for GitHub and AWS.
# Configure Access
Source: https://kubestarterkit.com/usage/getting-started/04-configure-access
Set up user access for GitHub and AWS
## Overview
After bootstrapping accounts, configure access for team members. This page covers:
* IAM Identity Center users and groups
* GitHub organization membership
The user performing the bootstrap already exists in IAM Identity Center and GitHub. You must import these existing resources before Terraform can manage them (import instructions below).
## Configure User Management
The kit manages both AWS IAM Identity Center and GitHub organization membership from a single `users.yaml` file via Terraform.
### Authenticate with GitHub CLI
The Terraform GitHub provider uses credentials from the GitHub CLI. Authenticate with an account that has **Owner** permissions on your organization:
```bash theme={null}
gh auth login
```
Follow the prompts to authenticate. Verify you have the necessary permissions:
```bash theme={null}
gh api orgs//memberships/$( gh api user --jq '.login' ) --jq '.role'
```
This should return `admin`.
### Add Users
Add users to `terraform/live/shared/global/user-management/data/users.yaml`:
```yaml theme={null}
users:
- github:
username: your-github-username
role: admin
teams:
Admin:
role: maintainer
aws:
user_name: you@example.com
email: you@example.com
group_membership: [Admin]
given_name: Your
family_name: Name
```
This file is the single source of truth for both GitHub and AWS access.
Import your existing IAM Identity Center user, groups, and GitHub membership. Add import blocks to `terraform/live/shared/global/user-management/imports.tf`:
```hcl theme={null}
# IAM Identity Center user
import {
to = module.aws-iam-identity-center.aws_identitystore_user.sso_users["you@example.com"]
id = "/"
}
# IAM Identity Center group (if it exists)
import {
to = module.aws-iam-identity-center.aws_identitystore_group.sso_groups["Admin"]
id = "/"
}
# IAM Identity Center group membership
import {
to = module.aws-iam-identity-center.aws_identitystore_group_membership.sso_group_membership["you@example.com_Admin"]
id = "/"
}
# GitHub organization member
import {
to = module.github_membership.github_membership.this["your-github-username"]
id = ":your-github-username"
}
```
To find the required AWS IDs:
```bash theme={null}
# Get the Identity Store ID
aws sso-admin list-instances --query 'Instances[0].IdentityStoreId' --output text
# List users
aws identitystore list-users --identity-store-id
# List groups
aws identitystore list-groups --identity-store-id
# Get a group membership ID
aws identitystore get-group-membership-id \
--identity-store-id \
--group-id \
--member-id UserId=
```
To find existing GitHub members:
```bash theme={null}
gh api orgs//members --jq '.[].login'
```
```bash theme={null}
cd terraform
terramate run --tags user-management -- terraform init
terramate run --tags user-management -- terraform apply
```
This creates/imports:
* IAM Identity Center users
* Group memberships (Admin, PowerUser, ReadOnly)
* Permission set assignments to all accounts
* GitHub organization memberships
## Verify Access Configuration
Before proceeding, verify that your existing users and groups are correctly configured in both AWS and GitHub.
### Review IAM Identity Center
1. Open the [IAM Identity Center console](https://console.aws.amazon.com/singlesignon/home) in your Management account
2. Navigate to **Users** and verify your bootstrap user exists
3. Navigate to **Groups** and verify the expected groups exist (Admin, PowerUser, ReadOnly)
4. Check group memberships by clicking on each group
### Review GitHub Organization
1. Open your GitHub organization's **People** page: `https://github.com/orgs//people`
2. Verify your bootstrap user appears as an **Owner**
3. Check the **Invitations** tab for pending invites; users who haven't accepted their invitation won't appear in the members list
Users with pending invitations only appear under the **Invitations** tab, not in the main members list. Make sure to check both when verifying membership.
## Groups and Permission Sets
The kit configures three groups with corresponding permission sets:
| Group | Permission Set | Access Level |
| --------- | ------------------- | --------------------------- |
| Admin | AdministratorAccess | Full access to all accounts |
| PowerUser | PowerUserAccess | Full access except IAM |
| ReadOnly | ViewOnlyAccess | Read-only access |
## Next Steps
With access configured, proceed to [Configure Integrations](/usage/getting-started/05-configure-integrations) to set up external service integrations for CI/CD.
# Configure Integrations
Source: https://kubestarterkit.com/usage/getting-started/05-configure-integrations
Configure AWS, GitHub, and Terramate Cloud integrations
## Overview
With [human access configured](/usage/getting-started/04-configure-access), you now need to configure the external service integrations that enable CI/CD automation. This page covers:
* **AWS**: Account IDs and role ARNs for Terraform
* **GitHub (octo-sts)**: Tokens for managing GitHub organization resources
* **Terramate Cloud** (optional): Plan visualization and drift detection
## Verify Terraform Configuration
Your `terraform/config.tm.hcl` should already have the correct values from the [Bootstrap Accounts](/usage/getting-started/03-bootstrap-accounts) step. Verify they look like this:
```hcl theme={null}
globals {
namespace = ""
# These were set during bootstrap
github_oidc_assume_role_arn = "arn:aws:iam:::role/-gbl-infra-bootstrap-github-oidc"
sso_admin_assume_role_arn = "arn:aws:iam:::role/aws-reserved/sso.amazonaws.com//AWSReservedSSO_AdministratorAccess_XXXXX"
backend_bucket = "-gbl-infra-bootstrap-state"
backend_region = ""
}
```
## Configure GitHub Repository Variables
GitHub Actions needs the OIDC role ARN to authenticate. Add these variables to your repository (Settings > Secrets and variables > Actions > Variables):
| Variable | Value |
| ------------------------ | ---------------------------------------------------------------------------------- |
| `TERRAFORM_AWS_ROLE_ARN` | `arn:aws:iam:::role/-gbl-infra-bootstrap-github-oidc` |
| `TERRAFORM_AWS_REGION` | Your primary AWS region (e.g., `us-east-2`) |
These are repository **variables**, not secrets, since the values are not sensitive. GitHub Actions only needs the Infrastructure account role. Cross-account access is handled by each Terraform stack's provider configuration.
## Update GitHub Workflows
The workflows read credentials from GitHub repository variables. Verify the workflow files reference the variables correctly:
```yaml theme={null}
# In .github/workflows/terramate-preview.yml, terramate-deploy.yml, and terramate-detect-drift.yml
- name: "Configure AWS Credentials"
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.TERRAFORM_AWS_REGION }}
role-to-assume: ${{ vars.TERRAFORM_AWS_ROLE_ARN }}
```
## Configure octo-sts for GitHub Tokens
The Terramate workflows use [octo-sts](https://github.com/octo-sts/app) to obtain GitHub tokens for managing GitHub organization resources (teams, members, repository settings). This is more secure than storing long-lived GitHub tokens as secrets.
1. Navigate to the [octo-sts GitHub App](https://github.com/apps/octo-sts)
2. Click **Install** and select your organization
3. Grant access to your forked repository (or all repositories)
Update each policy file in `.github/chainguard/` to reference your organization:
**`.github/chainguard/terramate.sts.yaml`**:
```yaml theme={null}
issuer: https://token.actions.githubusercontent.com
subject_pattern: "^repo:/:.*"
claim_pattern:
workflow_ref: '^//\.github/workflows/terramate.*@refs/.*$'
permissions:
administration: write
metadata: read
members: write
contents: read
```
**`.github/chainguard/release-please.sts.yaml`**:
```yaml theme={null}
issuer: https://token.actions.githubusercontent.com
subject: "repo:/:ref:refs/heads/main"
permissions:
contents: write
pull_requests: write
```
Replace `` with your GitHub organization name and `` with your repository name.
## Configure Terramate Cloud (Optional)
If you want to use [Terramate Cloud](https://cloud.terramate.io/) for plan visualization and drift detection:
1. Create an account at [cloud.terramate.io (EU)](https://cloud.terramate.io/) OR [us.cloud.terramate.io (US)](https://us.cloud.terramate.io/)
2. Install the GitHub App from your Terramate Cloud dashboard under **Integrations** to enable PR comments and status checks
3. Update the `cloud_organization` in `.github/workflows/terramate-preview.yml` and other Terramate workflows:
```yaml theme={null}
- name: Install Terramate
uses: terramate-io/terramate-action@0500f8a40b57a793a41edd2aea0a49e31c7204e8 # v3.2.0
with:
version: "0.15.1"
cloud_organization: your-terramate-org
```
Terramate Cloud is optional but recommended. It provides a unified view of Terraform plans across all stacks and makes PR reviews much easier.
## Search for Remaining References
Search for any remaining references to the original organization or AWS accounts:
```bash theme={null}
# Find references to the original organization
grep -r "DevOps-Directive" .github/
grep -r "094905625236" . # Original AWS account ID
```
## Verify Your Setup
```bash theme={null}
cd terraform
terramate fmt --check
terramate generate
```
Review the generated files to ensure your configuration changes propagated correctly.
## Commit Your Changes
```bash theme={null}
git add .
git commit -m "chore: configure AWS account IDs and role ARNs"
git push
```
## Verify CI/CD
Push a commit or open a pull request to verify that GitHub Actions can authenticate to AWS:
1. Check the workflow run in the **Actions** tab
2. The `Configure AWS Credentials` step should succeed
3. Terraform plan output should appear (if using Terramate Cloud, check the PR comments)
## Troubleshooting
### "Access Denied" when assuming cross-account role
The two-step authentication means there are two places trust can fail:
1. **GitHub → Infrastructure account**: Check the OIDC role trust policy
```bash theme={null}
# Run from Infrastructure account (replace with your namespace)
aws iam get-role --role-name "-gbl-infra-bootstrap-github-oidc" \
--query 'Role.AssumeRolePolicyDocument'
```
2. **Infrastructure account → Target account**: Check the target role trust policy
```bash theme={null}
# Run from target account (e.g. staging)
aws iam get-role --role-name "-gbl-staging-bootstrap-admin" \
--query 'Role.AssumeRolePolicyDocument'
```
The target account's trust policy should include the Infrastructure account's GitHub OIDC role ARN as a trusted principal.
### GitHub Actions can't authenticate
1. Verify the OIDC provider exists in the Infrastructure account:
```bash theme={null}
aws iam list-open-id-connect-providers
```
2. Check the GitHub OIDC role trust policy allows your repository:
```bash theme={null}
# Replace with your namespace
aws iam get-role --role-name "-gbl-infra-bootstrap-github-oidc" \
--query 'Role.AssumeRolePolicyDocument'
```
3. Ensure the repository name in the trust policy matches exactly (case-sensitive, including organization name).
## Next Steps
With AWS integration configured, you're ready to [Deploy Infrastructure](/usage/getting-started/06-deploy-infrastructure) to provision networking and EKS clusters.
# Deploy Infrastructure
Source: https://kubestarterkit.com/usage/getting-started/06-deploy-infrastructure
Deploy networking, EKS, and application resources via Terraform
## Overview
With accounts bootstrapped and integrations configured, you're ready to deploy the core AWS infrastructure. This includes networking (VPC, subnets, NAT), EKS clusters, and application-specific resources like S3 buckets and IAM roles.
### Decisions
Before deploying, review and customize these configuration options in each stack's `config.tm.hcl`:
#### Networking
| Setting | Location | Default | Description |
| -------------------- | ------------------------------------------------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------- |
| VPC CIDR | `globals.networking.vpc_cidr` | `10.0.0.0/16` | IP address range for the VPC. Use non-overlapping ranges if deploying multiple VPCs or connecting to on-premise networks. |
| NAT mode | `globals.networking.nat_mode` | `fck_nat` | How private subnets access the internet. See [NAT Gateway Modes](#nat-gateway-modes) below. |
| Bastion host | `globals.networking.enable_bastion` | `true` | Whether to create a bastion host for SSH/SSM access to private resources. |
| PlanetScale endpoint | `globals.networking.planetscale_endpoint_service_name` | Region-specific | VPC endpoint for PlanetScale private connectivity. Remove if not using PlanetScale. |
#### EKS Cluster
| Setting | Location | Default | Description |
| ------------------ | ------------------------------------------------ | -------------------- | ---------------------------------------------------------------------------------------------------------- |
| Kubernetes version | `globals.eks.kubernetes_version` | `1.34` | EKS control plane version. Update when upgrading clusters. |
| Node group version | `globals.eks.base_node_group_kubernetes_version` | `1.34` | Can lag control plane during rolling upgrades. |
| Public endpoint | `globals.eks.endpoint_public_access` | `false` | Whether the API server is publicly accessible. Set `false` for private-only access (requires bastion/VPN). |
| Private endpoint | `globals.eks.endpoint_private_access` | `true` | Whether the API server is accessible from within the VPC. |
| ArgoCD hostname | `globals.eks.argocd_hostname` | Environment-specific | FQDN for ArgoCD (used for webhook configuration). |
#### Hardcoded Values You May Want to Customize
These values are set in the Terraform modules and require editing the module source code to change:
| Setting | Location | Default | Description |
| ------------------------------ | ------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------ |
| Availability zones | `terraform/modules/*/main.tf` | First 3 AZs | Number of AZs used for subnets. Currently hardcoded to 3. |
| Base node group sizing | `terraform/modules/eks/main.tf` | 2-3 nodes | `min_size`, `max_size`, `desired_size` for the managed node group. |
| Base node group instance types | `terraform/modules/eks/variables.tf` | `t3.large` | Instance types for the initial managed node group. |
| Base node group AMI | `terraform/modules/eks/variables.tf` | `AL2023_x86_64_STANDARD` | AMI type (AL2023, Bottlerocket, etc.). |
| fck-nat instance type | `terraform/modules/networking/variables.tf` | `t4g.nano` | Instance size for fck-nat NAT instances. |
| Bastion instance type | `terraform/modules/networking/variables.tf` | `t4g.nano` | Instance size for the bastion host. |
| EKS addon versions | `terraform/modules/eks/variables.tf` | Pinned versions | Versions for CoreDNS, VPC CNI, kube-proxy, EBS CSI driver, Pod Identity agent. |
| SSO admin role ARN | `terraform/modules/eks/variables.tf` | Hardcoded | IAM Identity Center role granted cluster admin access. |
## Infrastructure Deployment Order
Terramate manages dependencies between stacks automatically. The deployment order is:
```
1. Networking → VPC, subnets, NAT gateway, bastion host
2. EKS → Kubernetes cluster, node groups, Karpenter
3. App Resources → S3 buckets, IAM roles for workloads
```
Each stack declares its dependencies, so Terramate applies them in the correct order.
## Initial Deployment (Local)
For the first deployment, you must run Terraform locally. The CI/CD workflow only applies *changed* stacks, and new stacks without any Terraform state aren't detected as changed.
Terramate can deploy all stacks at once with automatic dependency ordering (`terramate run --tags staging -- terraform apply`). However, deploying stacks sequentially makes it easier to follow progress, verify each component is working, and troubleshoot issues.
Use Leapp to start a session for the Infrastructure account:
```bash theme={null}
leapp session start "Infrastructure"
# Verify you're authenticated
aws sts get-caller-identity
```
```bash theme={null}
cd terraform
export REGION="us-east-2" # Your AWS region
export STAGE="staging" # or "prod"
# Initialize the networking stack
terramate run --tags ${STAGE}:${REGION}:networking -- terraform init
# Preview changes
terramate run --tags ${STAGE}:${REGION}:networking -- terraform plan
# Apply
terramate run --tags ${STAGE}:${REGION}:networking -- terraform apply
```
After networking is complete:
```bash theme={null}
# Initialize the EKS stack
terramate run --tags ${STAGE}:${REGION}:eks -- terraform init
# Preview and apply
terramate run --tags ${STAGE}:${REGION}:eks -- terraform plan
terramate run --tags ${STAGE}:${REGION}:eks -- terraform apply
```
EKS cluster creation takes 10-15 minutes.
If you have application-specific infrastructure:
```bash theme={null}
terramate run --tags ${STAGE}:${REGION}:services -- terraform init
terramate run --tags ${STAGE}:${REGION}:services -- terraform apply
```
After successful deployment, commit any updated `.terraform.lock.hcl` files:
```bash theme={null}
git add terraform/live/**/.terraform.lock.hcl
git commit -m "chore: update terraform lock files after initial deployment"
git push
```
## Subsequent Changes via Pull Request
After the initial deployment, use pull requests for all infrastructure changes. This ensures changes are reviewed and tracked.
Edit the relevant `config.tm.hcl` or module files, then regenerate:
```bash theme={null}
cd terraform
terramate generate
```
```bash theme={null}
git checkout -b infra/update-eks-version
git add .
git commit -m "chore: upgrade EKS to 1.34"
git push -u origin infra/update-eks-version
```
The CI workflow will:
1. Run `terraform plan` for each changed stack
2. Post plan output to Terramate Cloud (if configured)
3. Post a plan summary as a PR comment (if using Terramate Cloud)
Review the plan carefully before approving.
Once approved, merge the PR. The deploy workflow will apply changes in dependency order.
## What Gets Created
### Networking Stack
| Resource | Description |
| --------------- | ------------------------------------------------------------------------- |
| VPC | Isolated network with configurable CIDR (default: `10.0.0.0/16`) |
| Public subnets | 3 subnets across availability zones for load balancers |
| Private subnets | 3 subnets for EKS nodes and workloads + bastion |
| NAT gateway | Internet access for private subnets (fck-nat by default for cost savings) |
| S3 VPC endpoint | Free gateway endpoint for S3 access without NAT |
| Bastion host | EC2 instance for SSH tunneling to private resources |
### EKS Stack
| Resource | Description |
| ----------------------- | ----------------------------------------------- |
| EKS cluster | Managed Kubernetes control plane |
| Managed node group | Initial nodes for system workloads |
| Karpenter Prerequisites | Autoscaler for dynamic node provisioning |
| EBS CSI driver | Persistent volume support with encryption |
| Pod Identity | AWS IAM integration for workload authentication |
| CoreDNS, kube-proxy | Essential cluster add-ons |
### Application Resources (go-backend example)
| Resource | Description |
| ------------------------ | ----------------------------------------------- |
| S3 bucket | Application-specific storage |
| IAM role | Pod Identity role for AWS API access |
| Pod Identity association | Links the IAM role to Kubernetes ServiceAccount |
## Configuration Options
### NAT Gateway Modes
The networking module supports three NAT modes via `nat_mode` variable:
| Mode | Cost | Availability | Use Case |
| ------------------------ | ------------------ | ----------------------- | ------------------------------------------------ |
| `fck_nat` | \~\$5/month per AZ | HA with auto-failover | Development, staging |
| `single_nat_gateway` | \~\$45/month | Single point of failure | Cost-sensitive production |
| `one_nat_gateway_per_az` | \~\$135/month | Full HA | Production with strict availability requirements |
Configure in `terraform/live/staging//networking/config.tm.hcl`:
```hcl theme={null}
globals {
nat_mode = "fck_nat" # or "single_nat_gateway" or "one_nat_gateway_per_az"
}
```
### EKS Node Configuration
Karpenter handles most node provisioning, but you can configure the initial managed node group:
```hcl theme={null}
globals {
eks_managed_node_groups = {
system = {
instance_types = ["m6i.large"]
min_size = 2
max_size = 4
desired_size = 2
}
}
}
```
## Verify Deployment
After deployment completes:
```bash theme={null}
terramate run --tags ${STAGE}:${REGION}:eks -- terraform output
```
Note the `cluster_name` output; you'll need it for later steps.
Navigate to the [EKS console](https://console.aws.amazon.com/eks) and verify:
* Cluster status is `Active`
* Node group shows nodes in `Ready` state
* Add-ons (CoreDNS, kube-proxy, VPC CNI, EBS CSI) are `Active`
If you configured private-only endpoint access (`endpoint_public_access = false`), you cannot run `kubectl` commands from your local machine without first connecting through the bastion host. Console verification is sufficient for now; you'll configure cluster access via ArgoCD in the next step.
## Deploy Production
Production deployment follows the same pattern with production-specific configuration:
```bash theme={null}
# List production stacks
terramate list --tags prod:infrastructure
# Deploy locally (replace with your region)
terramate run --tags prod:${REGION}:networking -- terraform apply
terramate run --tags prod:${REGION}:eks -- terraform apply
```
## Next Steps
With infrastructure deployed, proceed to [Cluster Access](/usage/getting-started/07-cluster-access) to configure kubectl access to your EKS cluster.
# Set Up Cluster Access
Source: https://kubestarterkit.com/usage/getting-started/07-cluster-access
Connect to your EKS clusters via bastion and SOCKS proxy
## Overview
EKS clusters in Kube Starter Kit are configured with private API endpoints by default for security. This means you can't access the Kubernetes API directly from the internet; you need to go through the bastion host using a SOCKS5 proxy.
This page covers:
* Setting up SSH over AWS SSM Session Manager
* Connecting to the cluster via SOCKS proxy
* Configuring kubectl for persistent proxy access
If you prefer simpler access, you can enable the public API endpoint by setting `endpoint_public_access = true` in the EKS configuration (see [Deploy Infrastructure - EKS Cluster](/usage/getting-started/06-deploy-infrastructure#eks-cluster)). With a public endpoint, you can run `aws eks update-kubeconfig` and use kubectl directly without a proxy. However, this exposes your Kubernetes API to the internet, while still protected by IAM authentication, it increases your attack surface and may not meet compliance requirements.
## Architecture
```
Your Machine AWS VPC
┌───────────┐ ┌───────────────────────────┐
│ │ │ │
│ kubectl │───SOCKS5─────>│ Bastion Host │
│ │ proxy │ (private subnet) │
└───────────┘ │ │ │
│ │ ▼ │
│ SSM Session │ ┌───────────────┐ │
└────────────────────>│ │ EKS API │ │
(via AWS APIs) │ │ (private) │ │
│ └───────────────┘ │
└───────────────────────────┘
```
The bastion host:
* Lives in a private subnet (no public IP)
* Uses AWS SSM Session Manager for access (no SSH keys to manage)
* Acts as a [SOCKS5](https://en.wikipedia.org/wiki/SOCKS) proxy for kubectl traffic
## One-Time Setup
If you haven't already, run `mise install` to install the required tools (AWS CLI, Session Manager plugin, kubectl).
### Configure SSH for SSM
Add the SSM proxy configuration to your SSH config:
```bash theme={null}
# View the required config
mise run //tools:bastion:setup-ssh-config
```
Add this to `~/.ssh/config`:
```
# AWS SSM Session Manager SSH proxy
Host i-* mi-*
User ec2-user
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
```
This allows SSH to instances via SSM using the instance ID as the hostname.
## Connect to the Cluster
Start a Leapp session for the target account:
```bash theme={null}
leapp session start "Staging"
```
Verify authentication:
```bash theme={null}
aws sts get-caller-identity
```
Get the cluster credentials:
```bash theme={null}
mise run //tools:eks:get-credentials {cluster-name}
```
Or manually:
```bash theme={null}
export REGION="us-east-2" # Your AWS region
aws eks update-kubeconfig \
--name ${CLUSTER_NAME} \
--region ${REGION} \
--alias staging
```
In a **separate terminal** (with the same Leapp session active), start the proxy:
```bash theme={null}
mise run //tools:eks:connect staging
```
This automatically looks up the bastion instance and starts a SOCKS5 proxy on `localhost:1080`. Keep this terminal open while accessing the cluster.
The task automatically pushes your SSH public key via EC2 Instance Connect (valid for 60 seconds) before establishing the SSH tunnel.
**Option A: Per-command (temporary)**
```bash theme={null}
HTTPS_PROXY=socks5://localhost:1080 kubectl get nodes
```
**Option B: Update kubeconfig (persistent)**
```bash theme={null}
kubectl config set-cluster {cluster-name} --proxy-url=socks5://localhost:1080
```
Now kubectl commands work without the environment variable:
```bash theme={null}
kubectl get nodes
kubectl get pods -A
```
## Configure Persistent Access
To avoid passing the proxy URL each time, configure it in your kubeconfig:
Set the proxy for a specific cluster context:
```bash theme={null}
kubectl config set-cluster staging --proxy-url=socks5://localhost:1080
```
This modifies `~/.kube/config` to include the proxy URL for that cluster.
Set the proxy globally for all kubectl commands:
```bash theme={null}
export HTTPS_PROXY=socks5://localhost:1080
kubectl get nodes
```
Add to your shell profile for persistence (but note this affects all HTTPS traffic).
## Production Access
The same process applies to production:
```bash theme={null}
# Authenticate to production
leapp session start "Production"
# Update kubeconfig
mise run //tools:eks:get-credentials {cluster-name}
# Start proxy (in separate terminal)
mise run //tools:eks:connect production
# Configure persistent proxy
kubectl config set-cluster {cluster-name} --proxy-url=socks5://localhost:1080
# Use kubectl
kubectl get nodes
```
For production access, consider implementing additional access controls:
* Require MFA for SSM sessions
* Use AWS CloudTrail to audit access
* Implement just-in-time access with temporary permissions
## Next Steps
With cluster access configured, proceed to [Deploy Kubernetes Baseline](/usage/getting-started/08-deploy-kubernetes-baseline) to bootstrap ArgoCD and deploy infrastructure components.
# Deploy Kubernetes Baseline
Source: https://kubestarterkit.com/usage/getting-started/08-deploy-kubernetes-baseline
Bootstrap ArgoCD and deploy infrastructure components
## Overview
With EKS clusters deployed and [cluster access configured](/usage/getting-started/07-cluster-access), the next step is to bootstrap ArgoCD and deploy the Kubernetes baseline components. ArgoCD uses an app-of-apps pattern to manage all cluster resources declaratively.
Before proceeding, ensure you have `kubectl` access to the cluster. See [Cluster Access](/usage/getting-started/07-cluster-access) for setup instructions.
## Architecture
The kit uses a three-tier app-of-apps pattern:
```
argocd-app-of-apps (root)
├── argocd → Self-manages ArgoCD installation
├── infrastructure-app-of-apps → Manages infrastructure components
│ ├── cert-manager
│ ├── traefik
│ ├── external-secrets
│ ├── external-dns
│ ├── karpenter
│ ├── cloudnative-pg
│ ├── signoz-k8s-infra
│ └── reloader
└── services-app-of-apps → Manages application deployments
├── go-backend
└── go-backend-helm
```
Each ArgoCD Application points to rendered manifests in `kubernetes/rendered/{cluster}/`.
## Update Repository URLs
Before deploying, update the Git repository URLs to point to your fork:
The Application manifests reference the Git repository. Update the `repoURL` in the source templates:
**`kubernetes/src/argocd/argocd/templates/Application.argocd-app-of-apps.yaml`:**
```yaml theme={null}
spec:
source:
repoURL: "git@github.com:/.git" # Update this
targetRevision: main
path: kubernetes/rendered/{{ .Values.cluster }}/argocd/argocd
```
Update this URL in all Application templates:
* `kubernetes/src/argocd/argocd/templates/Application.*.yaml`
* `kubernetes/src/argocd/infrastructure/templates/Application.GENERATOR.yaml`
* `kubernetes/src/argocd/services/templates/Application.GENERATOR.yaml`
After updating the source files, render the manifests for your cluster:
```bash theme={null}
mise run //kubernetes/src/argocd:render-all staging
mise run //kubernetes/src/infrastructure:render-all staging
mise run //kubernetes/src/services:render-all staging
```
Replace `staging` with `production` for production clusters.
```bash theme={null}
git add .
git commit -m "chore: update ArgoCD repository URLs for fork"
git push
```
## Create Deploy Key
ArgoCD needs SSH access to clone your private repository. Run the bootstrap task to generate a deploy key and create the Kubernetes secret:
```bash theme={null}
cd kubernetes/bootstrap
mise run create-deploy-key "git@github.com:/.git"
```
The command outputs the public key. Add it to your repository:
1. Go to your repository **Settings → Deploy keys**
2. Click **Add deploy key**
3. Title: `ArgoCD Deploy Key`
4. Paste the public key from the command output
5. Leave **Allow write access** unchecked (read-only is sufficient for GitOps)
6. Click **Add key**
The private key is stored as a Kubernetes Secret (`repo-kube-starter-kit`) in the `argocd` namespace. ArgoCD automatically discovers it via the `argocd.argoproj.io/secret-type=repository` label.
## Configure GitHub OAuth (Optional)
If you want to enable GitHub OAuth for ArgoCD login (recommended for production), you need to store the OAuth app secret in AWS Secrets Manager.
1. Go to your GitHub organization **Settings → Developer settings → OAuth Apps**
2. Click **New OAuth App**
3. Fill in the details:
* **Application name**: `ArgoCD Staging`
* **Homepage URL**: `https://argocd.staging.`
* **Authorization callback URL**: `https://argocd.staging./api/dex/callback`
4. Click **Register application**
5. Generate a new client secret and save both the Client ID and Client Secret
```bash theme={null}
export REGION="us-east-2" # Your AWS region
export CLUSTER_NAME=""
aws secretsmanager create-secret \
--name "${CLUSTER_NAME}-argocd-github-dex" \
--secret-string '{"clientId":"YOUR_CLIENT_ID","clientSecret":"YOUR_CLIENT_SECRET"}' \
--region ${REGION}
```
See the [ArgoCD Dex documentation](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#dex) for more details on SSO configuration.
## Bootstrap ArgoCD
Only ArgoCD itself needs to be manually deployed. Once running, ArgoCD will deploy and manage all other infrastructure components automatically.
```bash theme={null}
cd kubernetes/bootstrap
mise run install-argocd staging
```
This task:
1. Applies ArgoCD manifests (handles CRD ordering automatically)
2. Waits for ArgoCD deployments to be ready
3. Applies AppProjects
4. Applies Applications
ArgoCD will then automatically deploy all infrastructure components (cert-manager, external-secrets, external-dns, traefik, etc.) and services.
## Verify Bootstrap
The deploy key secret should have been created by the bootstrap task:
```bash theme={null}
kubectl get secrets -n argocd | grep repo-kube-starter-kit
```
Check the ArgoCD Applications:
```bash theme={null}
kubectl get applications -n argocd
```
You should see:
* `argocd-app-of-apps` - Synced
* `infrastructure-app-of-apps` - Synced
* `services-app-of-apps` - Synced
From here, ArgoCD manages everything automatically.
It may take several minutes for all applications to become healthy as reconciliation loops complete. During this time:
* Infrastructure components are deployed and start up
* External Secrets syncs secrets from AWS Secrets Manager
* Pods restart to pick up new secrets/configurations
* External DNS creates DNS records
* Cert Manager issues TLS certificates
You can monitor progress in the ArgoCD UI or by running `kubectl get applications -n argocd`.
## Access ArgoCD UI
Once external-dns creates the DNS records, ArgoCD is accessible at `https://argocd..` (e.g., `https://argocd.staging.example.com`).
Alternatively, use port-forwarding for immediate access:
```bash theme={null}
kubectl port-forward svc/argocd-server -n argocd 8080:443
```
Open [https://localhost:8080](https://localhost:8080) in your browser.
### Login
If you configured GitHub OAuth, use the **Log in via GitHub** button.
Otherwise, use the admin password:
```bash theme={null}
kubectl get secret argocd-initial-admin-secret -n argocd \
-o jsonpath="{.data.password}" | base64 -d
```
Login with username `admin` and the retrieved password.
The admin user is enabled by default for initial bootstrap and debugging. Once you've configured SSO (GitHub OAuth), consider disabling the admin user by setting `admin.enabled: "false"` in `kubernetes/src/infrastructure/argocd/values.yaml`.
## Infrastructure Components
The following components are included in the Kubernetes baseline:
| Component | Bootstrap | Purpose |
| ---------------------------- | --------- | ----------------------------------------------- |
| **argocd** | Manual | GitOps continuous delivery |
| **cert-manager** | ArgoCD | TLS certificate automation with Let's Encrypt |
| **external-secrets** | ArgoCD | Syncs secrets from AWS Secrets Manager |
| **external-dns** | ArgoCD | Automatic DNS record management in Route53 |
| **traefik** | ArgoCD | Ingress controller for routing external traffic |
| **karpenter** | ArgoCD | Dynamic node provisioning and autoscaling |
| **cloudnative-pg** | ArgoCD | PostgreSQL operator for in-cluster databases |
| **signoz-k8s-infra** | ArgoCD | OpenTelemetry collectors for observability |
| **reloader** | ArgoCD | Restarts pods when ConfigMaps/Secrets change |
| **ebs-csi-driver-resources** | ArgoCD | StorageClass for encrypted EBS volumes |
Only ArgoCD is deployed manually during bootstrap. All other components are deployed automatically by ArgoCD once it syncs with the repository.
## Verify Deployment
```bash theme={null}
kubectl get applications -n argocd
```
All applications should show `Synced` and `Healthy` status.
```bash theme={null}
kubectl get pods -n cert-manager
kubectl get pods -n traefik
kubectl get pods -n external-secrets
kubectl get pods -n karpenter
```
All pods should be `Running`.
```bash theme={null}
kubectl get ingress -A
```
Ingresses should have an ADDRESS assigned (the load balancer DNS name).
Once external-dns is running, check Route53 for new records:
```bash theme={null}
aws route53 list-resource-record-sets \
--hosted-zone-id "" \
--query "ResourceRecordSets[?Type=='A' || Type=='CNAME']"
```
## Enable/Disable Components
To enable or disable infrastructure components, edit `kubernetes/src/argocd/infrastructure/values.yaml`:
```yaml theme={null}
applications:
argocd:
enabled: true
cert-manager:
enabled: true
cloudnative-pg:
enabled: true
envoy-gateway:
enabled: false # Disabled by default
# ... other components
```
After changing, render and push:
```bash theme={null}
mise run //kubernetes/src/infrastructure:render-all ""
git add . && git commit -m "chore: enable/disable components"
git push
```
ArgoCD will automatically sync the changes.
## Next Steps
With the Kubernetes baseline deployed, proceed to [Local Development Setup](/usage/getting-started/09-local-development-setup) to set up KinD, Tilt, and mirrord for local development.
# Local Development Setup
Source: https://kubestarterkit.com/usage/getting-started/09-local-development-setup
Set up KinD, Tilt, and mirrord for local development
## Overview
The kit includes a complete local development environment using:
* **[KinD](https://kind.sigs.k8s.io/)** - Kubernetes in Docker for a local multi-node cluster
* **[Tilt](https://tilt.dev/)** - Live development environment with hot reloading
* **[mirrord](https://mirrord.dev/)** - Remote debugging by intercepting pod traffic
This lets you develop and test Kubernetes deployments locally before pushing to staging.
## Prerequisites
Ensure you have:
* A container runtime (Docker, Podman, or OrbStack)
* mise installed with tools (`mise install` from [Repository Setup](/usage/getting-started/02-repository-setup))
## Quick Start
```bash theme={null}
mise run //local:create-cluster
```
This creates a 3-node KinD cluster (1 control-plane, 2 workers) with a local Docker registry.
In a **separate terminal** (requires sudo):
```bash theme={null}
mise run //local:cloud-provider-kind
```
This enables LoadBalancer services to get IPs in the `172.20.0.0/16` range.
```bash theme={null}
mise run //local:tilt-up
```
Open [http://localhost:10350](http://localhost:10350) to view the Tilt dashboard.
```bash theme={null}
mise run //local:ingress-urls
```
This prints the sslip.io URLs for accessing services:
* `http://go-backend-kustomize.172-20-0-100.sslip.io`
* `http://go-backend-helm.172-20-0-100.sslip.io`
* `http://go-backend-timoni.172-20-0-100.sslip.io`
## What Gets Deployed
Tilt deploys a subset of infrastructure and all demo services:
### Infrastructure Components
| Component | Purpose |
| ------------------ | ---------------------------------------------------------- |
| **cert-manager** | TLS certificates (using self-signed ClusterIssuer locally) |
| **cloudnative-pg** | PostgreSQL operator for in-cluster databases |
| **traefik** | Ingress controller |
### Demo Services
The kit includes three variants of `go-backend`, each demonstrating a different Kubernetes templating approach:
| Service | Templating | Description |
| ----------------- | ---------- | ---------------------------------------------------- |
| go-backend | Kustomize | Uses overlays for environment-specific configuration |
| go-backend-helm | Helm | Traditional Helm chart with values files |
| go-backend-timoni | Timoni | CUE-based configuration with type safety |
All three deploy the same Go application with:
* PostgreSQL database (via CloudNativePG)
* Database migrations (via Atlas, run as Kubernetes Job)
* Ingress for external access
## Cluster Configuration
### KinD Cluster
The cluster is configured in `local/kind/cluster-config.yaml`:
```yaml theme={null}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
```
Key features:
* 3 nodes for realistic scheduling
* Local registry on `localhost:5001`
* Stable Docker network (`172.20.0.0/16`) for predictable IPs
### Local Registry
The `create-cluster` task creates a local Docker registry:
```bash theme={null}
# Images pushed to localhost:5001 are available in the cluster
docker build -t localhost:5001/my-app:latest .
docker push localhost:5001/my-app:latest
```
Tilt automatically builds and pushes images to this registry.
## Tilt Configuration
### Tiltfile Structure
```
local/tilt/Tiltfile # Main entrypoint
├── kubernetes/src/infrastructure/Tiltfile # Infrastructure components
└── kubernetes/src/services/Tiltfile # Application services
```
### Infrastructure Tiltfile
Deploys cluster components in order:
```python theme={null}
# 1. cert-manager (no dependencies)
# 2. cloudnative-pg (depends on cert-manager)
# 3. traefik (no dependencies)
```
Each component uses Helm with local values (`values.local.yaml`).
### Services Tiltfile
For each service:
1. Builds the Docker image
2. Waits for CloudNativePG to be ready
3. Creates a PostgreSQL Cluster CR
4. Runs database migrations
5. Deploys the application
### Customizing Tilt Behavior
Edit the values files for local overrides:
```bash theme={null}
# Infrastructure
kubernetes/src/infrastructure/*/values.local.yaml
# Services
kubernetes/src/services/*/values.local.yaml
```
## Developing Outside the Container with mirrord
While Tilt runs your services inside containers in the KinD cluster, [mirrord](https://mirrord.dev/) lets you run a service directly on your host machine while staying connected to the cluster. This is ideal when you want to iterate on application code without waiting for container rebuilds:
* **Skip container builds:** Just recompile and restart, no Docker builds or image pushes
* **Use your local tools:** Run with your IDE, debugger, profiler, or any host-native tooling
* **Access cluster resources:** Your local process can reach cluster-internal services (databases, APIs)
* **Inherit pod environment:** Environment variables and secrets from the target pod are available locally
### Configuration
The mirrord config is in `services/go-backend/.mirrord/mirrord.json`:
```json theme={null}
{
"target": { "path": "deployment/go-backend" },
"feature": {
"network": { "incoming": "steal", "outgoing": true },
"fs": "local",
"env": true
}
}
```
This configuration:
* **Steals** incoming traffic from the `go-backend` deployment in the KinD cluster
* Enables **outgoing** connections to cluster resources (database, etc.)
* Uses **local** filesystem
* Mirrors pod **environment variables** to your local process
### Using mirrord
The service must be deployed in the local cluster (via Tilt) before mirrord can intercept its traffic.
```bash theme={null}
cd services/go-backend
mirrord exec -- go run ./cmd/main.go
```
Your local process now receives traffic that would go to the in-cluster pod.
Use your IDE's debugger, add print statements, or modify code. Changes take effect immediately without rebuilding containers.
Most IDEs have mirrord plugins that integrate debugging directly. See the [mirrord IDE docs](https://mirrord.dev/docs/ide-plugins/) for VS Code, IntelliJ, and others.
## Comparison: Local vs Production
| Aspect | Local (KinD + Tilt) | Production (EKS + ArgoCD) |
| ---------- | ------------------- | ------------------------- |
| Cluster | KinD (Docker) | EKS (AWS) |
| Registry | localhost:5001 | ECR |
| Ingress | sslip.io | Route53 + real domain |
| TLS | Self-signed | Let's Encrypt |
| Secrets | Local values | AWS Secrets Manager |
| Database | CloudNativePG | CloudNativePG (or RDS) |
| Deployment | Tilt (live reload) | ArgoCD (GitOps) |
## Next Steps
With local development set up, you're ready to start building! See the [Operations](/usage/operations) guides for:
* [Updating 1st Party Applications](/usage/operations/02-updating-1st-party-applications) - Modify and deploy your services
* [Bootstrapping a New Service](/usage/operations/04-bootstrapping-new-service) - Add a new application
* [Database Operations](/usage/operations/07-database-operations) - Manage migrations and databases
# Making Terraform Changes
Source: https://kubestarterkit.com/usage/operations/01-updating-terraform-infrastructure
How to make updates to infrastructure via Terraform
## Overview
Infrastructure changes flow through Terramate and GitHub. Whether you're modifying an existing stack, adding a new resource, or creating entirely new infrastructure, the workflow is the same: make changes locally, open a PR, review the plan, and merge to apply.
## Workflow
Edit the relevant files in `terraform/`. This might be:
* Stack configuration (`config.tm.hcl`, `inputs.tm.hcl`)
* Module code in `terraform/modules/`
* New stack definitions (`stack.tm.hcl`)
If you changed any Terramate configuration (`.tm.hcl` files), regenerate the Terraform files:
```bash theme={null}
cd terraform
terramate generate
```
This updates generated files like `_backend.tf`, `_provider.tf`, and `_main.tf`.
Test your changes before pushing:
```bash theme={null}
# List affected stacks
terramate list --changed
# Run validation
terramate run --changed -- terraform validate
# Preview plan (requires AWS credentials)
terramate script run --changed -- plan
```
Push your branch and open a PR. The CI workflow will:
1. Check Terramate formatting
2. Detect changed stacks
3. Run `terraform plan` for each affected stack
4. Sync preview results to Terramate Cloud
Check the plan output in:
* GitHub Actions logs
* PR comment with plan summary (if using Terramate Cloud)
* Terramate Cloud dashboard (for a unified view across stacks)
Look for:
* Expected resource changes (create, update, destroy)
* No unintended side effects
* Correct dependency ordering
Once approved, merge the PR. The deploy workflow will:
1. Detect changed stacks
2. Apply changes in dependency order
3. Sync deployment results to Terramate Cloud
Use GitHub branch protection rules to control who can deploy infrastructure changes. Consider requiring PR approvals, setting up CODEOWNERS for the `terraform/` directory, and requiring status checks to pass before merging.
## Common Tasks
### Adding a New Stack
1. Create directory under `terraform/live/{stage}/{region}/`:
```bash theme={null}
mkdir -p "terraform/live///my-new-stack"
```
2. Create `stack.tm.hcl`:
```hcl theme={null}
stack {
id = "--my-new-stack"
name = "my-new-stack"
description = "Description of what this stack does"
tags = ["", "", "my-new-stack"]
# Declare dependencies if needed
after = ["tag:::eks"]
}
```
Replace `` (e.g., `use2`) and `` (e.g., `us-east-2`) with your values.
3. Create `config.tm.hcl` with stack-specific globals
4. Create `main.tf` or use module mixins
5. Generate files:
```bash theme={null}
terramate generate
```
You may need to run `terramate generate` twice. The first pass generates `_outputs.tm.hcl` for some stacks (e.g., EKS), which is then used by dependent stacks in the second pass.
### Modifying a Module
1. Edit the module in `terraform/modules/{module-name}/`
2. Open a PR - Terramate will plan all stacks that use the module
3. Review plans across all affected environments
4. Merge to apply changes everywhere
### Targeting Specific Stacks
Use tags to run commands on specific stacks:
```bash theme={null}
export REGION="us-east-2" # Your AWS region
export STAGE="staging" # or "prod"
# Single stack
terramate run --tags ${STAGE}:${REGION}:eks -- terraform plan
# All stacks in an environment
terramate run --tags ${STAGE} -- terraform plan
# All networking stacks
terramate run --tags networking -- terraform plan
```
## Troubleshooting
### "Repository has untracked files"
Terramate requires a clean git state. Either stage your changes or use:
```bash theme={null}
terramate run --disable-safeguards=git-untracked -- terraform plan
```
### Missing dependency outputs
When a dependency stack hasn't been applied yet, Terramate uses mock values. This is expected during initial bootstrap. Apply stacks in dependency order:
```bash theme={null}
# Check the order
terramate list --run-order
# Apply in order
terramate script run -- deploy
```
### Regenerate after config changes
If you see drift between generated files and configuration:
```bash theme={null}
terramate generate
git diff # Review changes
```
## Configuration Reference
### Root Configuration
The `terraform/config.tm.hcl` defines globals inherited by all stacks:
```hcl theme={null}
globals {
namespace = "" # e.g., "ksk"
github_oidc_role_arn = "arn:aws:iam::..."
sso_admin_role_arn = "arn:aws:iam::..."
terraform_state_bucket = "-gbl-infra-bootstrap-state"
terraform_state_region = "" # e.g., "us-east-2"
terraform_version = ">= 1.10"
aws_provider_version = "~> 6.27"
}
```
### Terramate Project Configuration
The `terramate.tm.hcl` at the repository root configures Terramate features:
```hcl theme={null}
terramate {
required_version = ">= 0.10.0"
config {
cloud {
organization = "your-org"
location = "us"
}
experiments = [
"outputs-sharing",
"scripts",
"tmgen"
]
}
}
```
### Stack Definition
Each stack requires a `stack.tm.hcl`:
```hcl theme={null}
# terraform/live/staging//networking/stack.tm.hcl
stack {
id = "staging--networking"
name = "networking"
description = "VPC and networking for staging "
tags = ["staging", "", "networking", "infrastructure"]
# Declare dependencies if needed
after = ["tag:staging::eks"]
}
```
### Stack-Specific Configuration
Override globals in `config.tm.hcl` within each stack:
```hcl theme={null}
# terraform/live/staging//networking/config.tm.hcl
globals {
vpc_cidr = "10.0.0.0/16"
nat_mode = "fck_nat" # Cost-effective NAT for non-prod
}
```
### Outputs Sharing
Cross-stack dependencies without `terraform_remote_state`:
```hcl theme={null}
# In networking stack: outputs.tm.hcl
output "vpc_id" {
backend = "terraform"
value = module.networking.vpc_id
}
# In EKS stack: inputs.tm.hcl
input "vpc_id" {
backend = "terraform"
from_stack_id = "staging-use2-networking"
value = outputs.vpc_id.value
mock = "vpc-mock12345" # Used during initial bootstrap
}
```
The input becomes a regular Terraform variable, usable in your module:
```hcl theme={null}
module "eks" {
source = "../../../modules/eks"
vpc_id = var.vpc_id
# ...
}
```
### Terramate Scripts
The `terraform/scripts.tm.hcl` file defines reusable commands used by CI:
```hcl theme={null}
script "preview" {
description = "Plan with outputs sharing"
job {
commands = [
["terraform", "validate"],
["terraform", "plan", "-out", "out.tfplan", "-detailed-exitcode", "-lock=false", {
sync_preview = true
terraform_plan_file = "out.tfplan"
enable_sharing = true
}],
]
}
}
script "deploy" {
description = "Apply with outputs sharing"
job {
commands = [
["terraform", "apply", "-auto-approve", "-lock-timeout=5m", {
enable_sharing = true
}],
]
}
}
```
### Mixins
Code generation templates live in `terraform/imports/mixins/`. These generate common files (`_backend.tf`, `_provider.tf`, `_main.tf`) from templates, eliminating copy-paste between stacks.
# Updating 1st Party Applications
Source: https://kubestarterkit.com/usage/operations/02-updating-1st-party-applications
Deploy changes to your applications
## Overview
First-party applications are services you build and deploy, like the `go-backend` example in the kit. This page covers how to make changes and deploy them through the GitOps pipeline.
## Deployment Flow
Changes flow through staging first, then to production after verification:
```
┌────────────┐
│ STAGING │ Merge to main -> Build -> Update manifests -> ArgoCD syncs ─┐
└────────────┘ │
┌──────────────────── Verify Changes ───────────────────────┘
┌────────────┐ │
│ PRODUCTION │ └─> Merge Release PR -> Build -> Update manifests -> ArgoCD syncs
└────────────┘
```
Each step is detailed in the sections below.
## Make Code Changes
Make changes to your application in `services/`:
```bash theme={null}
# Example: modify the go-backend service
vim services/go-backend/cmd/main.go
```
Use the [local development environment](/usage/getting-started/09-local-development-setup) to test:
```bash theme={null}
mise run //local:tilt-up
```
Tilt automatically rebuilds and redeploys when you save changes.
```bash theme={null}
git add services/go-backend/
git commit -m "feat: add new endpoint for health checks"
git push origin main
```
## What Happens After Push (Staging)
### 1. CI Build Workflow
The CI workflow (`.github/workflows/ci-build-push.yml`):
1. Detects which services changed using path filters
2. Generates a version tag (e.g., `0.2.2-rc0029-g1234567`)
3. Builds the Docker image
4. Pushes to ECR
5. Triggers the GitOps workflow
### 2. GitOps Update Workflow
The GitOps workflow (`.github/workflows/gitops-update-manifests.yml`):
1. Updates the image tag in values files (e.g., `kubernetes/src/services/go-backend-helm/values.yaml`)
2. Renders the Kubernetes manifests
3. Commits and pushes to `main`
The image tag is identified by a comment marker:
```yaml theme={null}
image:
version: 0.2.2-rc0029-g1234567 # staging_services/go-backend
```
### 3. ArgoCD Sync
ArgoCD watches the repository and automatically syncs when manifests change. You can monitor the sync in the ArgoCD UI or CLI:
```bash theme={null}
argocd app get go-backend-helm
argocd app sync go-backend-helm
```
## Version Tagging Strategy
| Trigger | Tag Format | Example | Environment |
| --------------- | ------------------- | ----------------------- | -------------- |
| Push to `main` | `X.Y.Z-rcNNNN-gSHA` | `0.2.2-rc0029-g1234567` | Staging |
| Release tag | `X.Y.Z` | `0.2.3` | Production |
| Manual workflow | User-specified | `hotfix-123` | User-specified |
### Pre-release Versions (Staging)
When you push to `main`, the version is generated as:
* Base: last release tag + 1 patch (e.g., `0.2.2` → `0.2.3`)
* Suffix: `-rcNNNN-gSHA` where NNNN is commits since last tag
This ensures every commit has a unique, sortable version.
### Release Versions (Production)
To deploy to production, create a release tag:
```bash theme={null}
git tag "services/go-backend@0.2.3"
git push origin "services/go-backend@0.2.3"
```
This triggers the same build workflow but:
* Uses the exact version from the tag (`0.2.3`)
* Sets environment to `production`
## Deploy to Production
Ensure the change works correctly in staging before promoting to production.
The Release Please workflow (`.github/workflows/release-please.yml`) automatically creates a release PR when changes are pushed to `main`. The PR:
* Bumps the version based on conventional commit messages
* Updates the CHANGELOG
* Shows all changes since the last release
When you merge the release PR:
1. A GitHub release is created with the new version tag
2. The CI workflow (`.github/workflows/ci-build-push.yml`) builds and pushes the production image
3. The GitOps workflow (`.github/workflows/gitops-update-manifests.yml`) updates production manifests
4. ArgoCD syncs to the production cluster
You can also create a release manually by pushing a tag:
```bash theme={null}
git tag "services/go-backend@0.2.3"
git push origin "services/go-backend@0.2.3"
```
Watch the CI workflow and ArgoCD sync:
```bash theme={null}
# Check workflow status
gh run list --workflow=ci-build-push.yml
# Check ArgoCD
argocd app get go-backend-helm --grpc-web
```
## Manual Deployment
For debugging or hotfixes, you can trigger deployments manually:
Build and push an image without deploying:
```bash theme={null}
gh workflow run ci-build-push.yml \
-f service=services/go-backend \
-f version=hotfix-123
```
Deploy an already-built image to an environment:
```bash theme={null}
gh workflow run gitops-update-manifests.yml \
-f service=services/go-backend \
-f version=0.2.2-rc0029-g1234567 \
-f environment=staging
```
## Update Kubernetes Configuration
To change Kubernetes configuration (replicas, resources, etc.) without changing application code:
For Helm-based services, edit the values file:
```bash theme={null}
vim kubernetes/src/services/go-backend-helm/values.yaml
```
For Kustomize-based services, edit the overlay:
```bash theme={null}
vim kubernetes/src/services/go-backend/staging/kustomization.yaml
```
Example changes:
```yaml theme={null}
replicas: 3 # Increase replicas
resources:
requests:
memory: "256Mi"
cpu: "250m"
```
```bash theme={null}
mise run //kubernetes/src/services:render-all ""
```
```bash theme={null}
git add kubernetes/
git commit -m "chore: increase go-backend replicas to 3"
git push origin main
```
ArgoCD syncs the new configuration without rebuilding the image.
## Rollback
To rollback to a previous version:
Update the image tag to a previous version:
```bash theme={null}
# Edit values.yaml
vim kubernetes/src/services/go-backend-helm/values..yaml
# Change image version to previous tag
# image:
# version: 0.2.2-rc0028-g98b8c55
# Render and push
mise run //kubernetes/src/services:render-all ""
git add . && git commit -m "fix: rollback go-backend to 0.2.2-rc0028"
git push origin main
```
Because auto-sync is enabled by default for all ArgoCD Applications, you would need to disable auto-sync for the app-of-apps AND the corresponding Application for this change to not be reverted.
```bash theme={null}
# View history
argocd app history go-backend-helm
# Rollback to previous revision
argocd app rollback go-backend-helm ""
```
## Next Steps
* [Updating 3rd Party Applications](/usage/operations/03-updating-3rd-party-applications) - Update infrastructure components
* [Bootstrapping a New Service](/usage/operations/04-bootstrapping-new-service) - Add a new application
# Updating 3rd Party Applications
Source: https://kubestarterkit.com/usage/operations/03-updating-3rd-party-applications
Update infrastructure components like traefik, cert-manager, and more
## Overview
Third-party applications are infrastructure components deployed via Helm charts, things like traefik, cert-manager, external-secrets, and ArgoCD itself. This page covers how to update their versions and configurations.
## Infrastructure Components
The kit includes these third-party components in `kubernetes/src/infrastructure/`:
| Component | Chart Source | Purpose |
| ---------------- | -------------------------- | ----------------------------- |
| argocd | argoproj.github.io | GitOps controller |
| cert-manager | quay.io/jetstack | TLS certificate automation |
| cloudnative-pg | cloudnative-pg.github.io | PostgreSQL operator |
| external-dns | kubernetes-sigs.github.io | DNS record management |
| external-secrets | charts.external-secrets.io | Secret synchronization |
| traefik | ghcr.io/traefik/helm | Ingress controller |
| karpenter | public.ecr.aws/karpenter | Node autoscaling |
| reloader | stakater.github.io | Pod restart on config changes |
| signoz-k8s-infra | charts.signoz.io | Observability collectors |
## Update Chart Versions
Each component uses a "wrapper chart" pattern: a local Helm chart that includes the upstream chart as a dependency.
Check the current version in the component's `values.yaml`:
```bash theme={null}
cat kubernetes/src/infrastructure/traefik/values.yaml
```
Look for the `chartVersions` section:
```yaml theme={null}
chartVersions:
traefik: "38.0.1"
```
Find the latest version from the chart repository:
```bash theme={null}
# Search for versions using OCI registry
helm search repo traefik/traefik --versions | head -10
```
Edit the values file to update the version:
```bash theme={null}
vim kubernetes/src/infrastructure/traefik/values.yaml
```
```yaml theme={null}
chartVersions:
traefik: "38.1.0" # Updated version
```
For environment-specific versions, edit `values.staging.yaml` or `values.production.yaml`. Setting these independently allows for testing upgrades in lower environments before rolling out to production.
Regenerate the lock file with the new dependency:
```bash theme={null}
cd kubernetes/src/infrastructure/traefik
helm dependency update
```
```bash theme={null}
mise run //kubernetes/src/infrastructure:render-all ""
git add .
git commit -m "chore: update traefik to 38.1.0"
git push origin main
```
## Update Configuration
To change component settings without upgrading versions:
Modify `values.yaml` for base configuration or `values.{environment}.yaml` for environment-specific settings:
```bash theme={null}
vim kubernetes/src/infrastructure/traefik/values.yaml
```
For example, increase controller replicas:
```yaml theme={null}
traefik:
deployment:
replicas: 3
```
```bash theme={null}
mise run //kubernetes/src/infrastructure:render-all ""
```
```bash theme={null}
git diff kubernetes/rendered/
git add .
git commit -m "chore: increase traefik replicas to 3"
git push origin main
```
ArgoCD automatically syncs the changes.
## Add Additional Manifests
Many components need additional manifests beyond the upstream chart (ClusterIssuers, StorageClasses, etc.). Add these in the wrapper chart's `templates/` directory:
```
kubernetes/src/infrastructure/cert-manager/
├── Chart.yaml
├── Chart.yaml.tmpl
├── values.yaml
├── values.staging.yaml
└── templates/
├── ClusterIssuer.letsencrypt-production.yaml
├── ClusterIssuer.letsencrypt-staging.yaml
└── ClusterIssuer.selfsigned.yaml
```
These templates are rendered alongside the upstream chart resources.
## Environment-Specific Configuration
Use values overlay files for environment differences:
```yaml theme={null}
# values.yaml (base)
traefik:
deployment:
replicas: 2
# values.staging.yaml
traefik:
deployment:
replicas: 1 # Smaller for staging
# values.production.yaml
traefik:
deployment:
replicas: 3 # Larger for production
```
The render process merges base values with environment-specific overrides.
## Enable/Disable Components
Components are enabled/disabled in the ArgoCD infrastructure app-of-apps:
```bash theme={null}
vim kubernetes/src/argocd/infrastructure/values.yaml
```
```yaml theme={null}
applications:
argocd:
enabled: true
cert-manager:
enabled: true
envoy-gateway:
enabled: false # Disabled
istio:
enabled: false # Disabled
```
After changing, render and push:
```bash theme={null}
mise run //kubernetes/src/infrastructure:render-all ""
git add . && git commit -m "chore: disable envoy-gateway"
git push origin main
```
## Upgrade Strategies
### Minor/Patch Updates
For minor and patch version updates (e.g., `4.12.2` → `4.12.3`):
1. Update the version in values.yaml
2. Run `helm dependency update`
3. Render, commit, and push
4. ArgoCD syncs automatically
### Major Updates
For major version updates (e.g., `4.x` → `5.x`):
1. **Read the changelog** for breaking changes
2. **Test in staging first** before production
3. **Update values** if configuration schema changed
4. **Have a rollback plan** if the component is critical
Major version upgrades may require CRD updates. Check the component's upgrade guide and apply CRD changes before upgrading the chart.
### CRD Updates
Some components (cert-manager, ArgoCD, Karpenter) use CRDs that may need manual updates:
```bash theme={null}
# Example: Update cert-manager CRDs
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.0/cert-manager.crds.yaml
```
Check the component's documentation for CRD upgrade procedures.
## Best Practices
1. **Test in staging first** - Always upgrade staging before production
2. **Read changelogs** - Especially for major versions
3. **Update one component at a time** - Easier to isolate issues
4. **Keep versions consistent** - Use the same version across environments when possible
5. **Monitor after upgrades** - Watch metrics and logs for regressions
## Next Steps
* [Adding K8s Infrastructure](/usage/operations/05-adding-k8s-infrastructure) - Add new infrastructure components
* [Managing Secrets](/usage/operations/06-managing-secrets) - Configure External Secrets
# Bootstrapping a New Service
Source: https://kubestarterkit.com/usage/operations/04-bootstrapping-new-service
Add a new application to the platform
## Overview
This guide walks through adding a new service to the platform. You'll create the application code, Dockerfile, Kubernetes manifests, and CI/CD configuration.
## Create the Service
```bash theme={null}
mkdir -p services/my-new-service
cd services/my-new-service
```
Create your application. For example, a Go service:
```bash theme={null}
go mod init my-new-service
mkdir cmd
```
Create `cmd/main.go`:
```go theme={null}
package main
import (
"log"
"net/http"
)
func main() {
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
})
log.Println("Starting server on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
```
Create `Dockerfile`:
```dockerfile theme={null}
FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o server ./cmd
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app/server /server
EXPOSE 8080
ENTRYPOINT ["/server"]
```
Create `mise.toml`:
```toml theme={null}
[env]
# Replace with your AWS account ID and region
IMAGE_REPO = ".dkr.ecr..amazonaws.com/services/my-new-service"
SERVICE_RELEASE_TAG = "services/my-new-service"
[tasks]
run = "go run ./cmd"
build-image = "docker build ."
```
```bash theme={null}
# Run the service
mise run run
# In another terminal, test it
curl http://localhost:8080/health
```
## Create ECR Repository
Add the new repository to the ECR configuration:
Edit `terraform/live/shared//ecr-repositories/main.tf` and add your service to the `local.repos` list:
```hcl theme={null}
locals {
region = var.aws_region
repos = [
"services/container-only-dummy",
"services/go-backend",
"services/go-backend/migrations",
"services/go-backend-no-migrations",
"services/my-new-service" # Add this
]
}
```
Commit and open a PR. The Terramate workflow will run a preview, and once merged, the ECR repository will be created.
Alternatively, apply directly:
```bash theme={null}
cd terraform
terramate run --tags ecr -- terraform apply
```
## Create Kubernetes Manifests
Choose your preferred templating approach. The kit demonstrates three options.
Create a Helm chart in `kubernetes/src/services/my-new-service/`:
```bash theme={null}
mkdir -p kubernetes/src/services/my-new-service/templates
```
**Chart.yaml:**
```yaml theme={null}
apiVersion: v2
name: my-new-service
version: 0.1.0
```
**values.yaml:**
```yaml theme={null}
replicas: 1
image:
# Replace with your AWS account ID and region
repository: ".dkr.ecr..amazonaws.com/services/my-new-service"
version: 0.1.0 # staging_services/my-new-service
ingress:
hostname: "my-new-service.staging."
```
**templates/deployment.yaml:**
```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-new-service
namespace: my-new-service
spec:
replicas: {{ .Values.replicas }}
selector:
matchLabels:
app: my-new-service
template:
metadata:
labels:
app: my-new-service
spec:
containers:
- name: my-new-service
image: "{{ .Values.image.repository }}:{{ .Values.image.version }}"
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /health
port: 8080
```
Add additional templates for Service, Ingress, etc.
Create a Kustomize structure in `kubernetes/src/services/my-new-service/`:
```bash theme={null}
mkdir -p kubernetes/src/services/my-new-service/{base,staging,production}
```
**base/kustomization.yaml:**
```yaml theme={null}
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
```
**staging/kustomization.yaml:**
```yaml theme={null}
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
images:
- name: my-new-service
# Replace with your AWS account ID and region
newName: ".dkr.ecr..amazonaws.com/services/my-new-service"
newTag: "0.1.0" # staging_services/my-new-service
```
## Add mise Tasks for Rendering
Create `kubernetes/src/services/my-new-service/mise.toml`:
```toml theme={null}
[tasks.render-cluster]
description = "Render manifests for a specific cluster"
usage = "arg '' help='Target cluster (staging, production)'"
run = '''
#!/usr/bin/env bash
set -euo pipefail
CLUSTER="${usage_cluster}"
OUTPUT_DIR="$(git rev-parse --show-toplevel)/kubernetes/rendered/${CLUSTER}/services/my-new-service"
mise run //tools:render:prep-output-dir "$OUTPUT_DIR"
helm template my-new-service . \
--namespace my-new-service \
--values values.yaml \
--values "values.${CLUSTER}.yaml" \
| mise run //tools:render:split-k8s-docs "$OUTPUT_DIR"
'''
```
## Register with ArgoCD
Add the service to the ArgoCD services app-of-apps:
Edit `kubernetes/src/argocd/services/values.yaml`:
```yaml theme={null}
applications:
go-backend:
enabled: true
go-backend-helm:
enabled: true
my-new-service:
enabled: true # Add this
```
```bash theme={null}
mise run //kubernetes/src/argocd:render-all "" # Adds the ArgoCD Application to the app-of-apps
mise run //kubernetes/src/services:render-all "" # Renders the service manifests
```
## Add to CI/CD
Edit `.github/utils/file-filters.yaml`:
```yaml theme={null}
services/my-new-service:
- 'services/my-new-service/**'
```
Edit `.github/workflows/ci-build-push.yml`:
```yaml theme={null}
inputs:
service:
options:
- services/go-backend
- services/my-new-service # Add this
```
Also update `.github/workflows/gitops-update-manifests.yml`.
## Test Locally
Edit `kubernetes/src/services/Tiltfile` to include your new service:
```python theme={null}
# Add your service
load_dynamic('./my-new-service/Tiltfile')
```
Create `kubernetes/src/services/my-new-service/Tiltfile` following the pattern of existing services.
```bash theme={null}
cd local
mise run tilt-up
```
Access via the sslip.io URL or port-forward:
```bash theme={null}
kubectl port-forward svc/my-new-service -n my-new-service 8080:80
curl http://localhost:8080/health
```
## Deploy to Staging
```bash theme={null}
git add .
git commit -m "feat: add my-new-service"
git push origin main
```
1. Watch the CI workflow build the image
2. Watch the GitOps workflow update manifests
3. Check ArgoCD for the new Application
## Next Steps
* [Managing Secrets](/usage/operations/06-managing-secrets) - Add secrets for your service
* [Database Operations](/usage/operations/07-database-operations) - Add a database if needed
* [Observability](/usage/operations/08-observability) - Configure metrics and tracing
# Adding New K8s Infrastructure Components
Source: https://kubestarterkit.com/usage/operations/05-adding-k8s-infrastructure
Add new infrastructure components and manage environment-specific configurations
## Overview
This guide covers adding new Kubernetes infrastructure components and managing configurations across environments. Infrastructure components are third-party tools deployed via Helm that provide platform capabilities.
## Add a New Infrastructure Component
```bash theme={null}
mkdir -p kubernetes/src/infrastructure/my-component/templates
cd kubernetes/src/infrastructure/my-component
```
The template allows environment-specific chart versions:
```yaml theme={null}
apiVersion: v2
name: my-component
version: 0.1.0
dependencies:
- name: my-component
version: "1.0.0" # Placeholder, replaced during render
repository: https://charts.example.com
```
Define base configuration and chart versions:
```yaml theme={null}
# Chart dependency versions
chartVersions:
my-component: "1.2.3"
# Pass-through values to the upstream chart
my-component:
replicaCount: 2
resources:
requests:
memory: "128Mi"
cpu: "100m"
```
**values.staging.yaml:**
```yaml theme={null}
chartVersions:
my-component: "1.2.3"
my-component:
replicaCount: 1
```
**values.production.yaml:**
```yaml theme={null}
chartVersions:
my-component: "1.2.3"
my-component:
replicaCount: 3
```
**values.local.yaml:**
```yaml theme={null}
my-component:
replicaCount: 1
```
Create templates in `templates/` for resources not provided by the upstream chart:
```yaml theme={null}
# templates/ServiceMonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-component
namespace: {{ .Release.Namespace }}
spec:
selector:
matchLabels:
app: my-component
endpoints:
- port: metrics
```
```toml theme={null}
[tasks.render-cluster]
description = "Render manifests for a specific cluster"
usage = 'arg "" help="Target cluster (staging, production)"'
run = '''
#!/usr/bin/env bash
set -euo pipefail
CLUSTER="${usage_cluster}"
OUTPUT_DIR="$(git rev-parse --show-toplevel)/kubernetes/rendered/${CLUSTER}/infrastructure/my-component"
# Inject chart versions from values files
VALUES_ENV="$CLUSTER" mise run //tools:render:inject-chart-versions
# Build dependencies
mise run //tools:render:helm-dep-build
# Prepare output directory
mise run //tools:render:prep-output-dir "$OUTPUT_DIR"
# Render and split into individual files
helm template my-component . \
--namespace my-component \
--values values.yaml \
--values "values.${CLUSTER}.yaml" \
| mise run //tools:render:split-k8s-docs "$OUTPUT_DIR"
'''
```
## Register with ArgoCD
Edit `kubernetes/src/argocd/infrastructure/values.yaml`:
```yaml theme={null}
applications:
argocd:
enabled: true
cert-manager:
enabled: true
my-component:
enabled: true # Add this
```
```bash theme={null}
mise run //kubernetes/src/argocd:render-all "" # Adds the ArgoCD Application to the app-of-apps
mise run //kubernetes/src/infrastructure:render-all "" # Renders the component manifests
```
```bash theme={null}
git add .
git commit -m "feat: add my-component infrastructure"
git push origin main
```
## Environment-Specific Configuration
### Values Hierarchy
Values are merged in order (later files override earlier):
1. `values.yaml` - Base configuration
2. `values.{cluster}.yaml` - Environment-specific overrides
### Common Patterns
**Different resource limits per environment:**
```yaml theme={null}
# values.yaml (base)
my-component:
resources:
requests:
memory: "256Mi"
cpu: "250m"
# values.staging.yaml
my-component:
resources:
requests:
memory: "128Mi"
cpu: "100m"
# values.production.yaml
my-component:
resources:
requests:
memory: "512Mi"
cpu: "500m"
```
**Different hostnames:**
```yaml theme={null}
# values.staging.yaml
my-component:
ingress:
host: my-component.staging.example.com
# values.production.yaml
my-component:
ingress:
host: my-component.example.com
```
**Feature flags:**
```yaml theme={null}
# values.staging.yaml
my-component:
debug: true
metrics:
enabled: true
# values.production.yaml
my-component:
debug: false
metrics:
enabled: true
```
## Add to Local Development
Create `values.local.yaml` with local-specific settings:
```yaml theme={null}
my-component:
replicaCount: 1
ingress:
host: my-component.127-0-0-1.sslip.io
```
Edit `kubernetes/src/infrastructure/Tiltfile`:
```python theme={null}
# Add my-component
k8s_yaml(
helm(
'my-component',
name='my-component',
namespace='my-component',
values=['my-component/values.yaml', 'my-component/values.local.yaml'],
)
)
k8s_resource(
'my-component',
resource_deps=['cert-manager'], # Add dependencies if needed
)
```
## Update the Render Pipeline
If your component needs to be rendered as part of the main render task:
Edit `kubernetes/mise.toml` to include your component:
```toml theme={null}
[tasks.render]
description = "Render all manifests for all environments"
run = '''
#!/usr/bin/env bash
set -euo pipefail
for cluster in staging production; do
# ... existing components ...
# Add your component
mise run //kubernetes/src/infrastructure/my-component:render-cluster "$cluster"
done
'''
```
## Namespace Management
Components typically run in their own namespace. Create it in your templates:
```yaml theme={null}
# templates/Namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Release.Namespace }}
labels:
app.kubernetes.io/name: my-component
```
Or include it in the Helm install:
```bash theme={null}
helm template my-component . \
--namespace my-component \
--create-namespace \
...
```
## Troubleshooting
### Helm dependency errors
```bash theme={null}
# Clear cached dependencies
rm -rf charts/ Chart.lock
# Re-add the repository
helm repo add my-repo https://charts.example.com
helm repo update
# Rebuild dependencies
helm dependency build
```
### Template rendering errors
```bash theme={null}
# Debug with verbose output
helm template my-component . \
--debug \
--values values.yaml \
--values values.staging.yaml
```
### ArgoCD not detecting the new Application
1. Verify the Application manifest was generated:
```bash theme={null}
ls kubernetes/rendered/staging/argocd/infrastructure/
```
2. Check the infrastructure app-of-apps is synced:
```bash theme={null}
argocd app get infrastructure-app-of-apps
```
3. Force a sync:
```bash theme={null}
argocd app sync infrastructure-app-of-apps
```
## Best Practices
1. **Use the wrapper chart pattern** - Never modify upstream charts directly
2. **Version pin dependencies** - Always specify exact versions in `chartVersions`
3. **Minimize environment differences** - Keep staging similar to production
4. **Document custom resources** - Explain why custom templates are needed
5. **Test locally first** - Use Tilt before deploying to staging
## Next Steps
* [Managing Secrets](/usage/operations/06-managing-secrets) - Configure secrets for your component
* [Updating 3rd Party Applications](/usage/operations/03-updating-3rd-party-applications) - Update component versions
# Managing Secrets
Source: https://kubestarterkit.com/usage/operations/06-managing-secrets
Create and manage secrets with External Secrets and AWS Secrets Manager
## Overview
The kit uses [External Secrets Operator](https://external-secrets.io/) to sync secrets from AWS Secrets Manager into Kubernetes. This keeps sensitive data out of Git while providing a GitOps-friendly secret management workflow.
## Architecture
```
AWS Secrets Manager Kubernetes Cluster
┌───────────────────┐ ┌────────────────────────────┐
│ │ │ │
│ ksk-use2-staging- │ │ ClusterSecretStore │
│ myapp-db │<───────│ (aws-secrets-manager) │
│ │ │ │ │
└───────────────────┘ │ ▼ │
│ ExternalSecret │
│ (myapp-secrets) │
│ │ │
│ ▼ │
│ Secret │
│ (myapp-secrets) │
│ │
└────────────────────────────┘
```
1. **ClusterSecretStore** - Configures access to AWS Secrets Manager (one per cluster)
2. **ExternalSecret** - Defines which secrets to fetch and how to map them
3. **Secret** - The Kubernetes Secret created and kept in sync
## Prerequisites
The External Secrets Operator uses Pod Identity to authenticate to AWS. This is configured automatically by the EKS Terraform module.
Verify the ClusterSecretStore is working:
```bash theme={null}
kubectl get clustersecretstores
kubectl describe clustersecretstore aws-secrets-manager
```
## Create a Secret in AWS
Using AWS CLI:
```bash theme={null}
export REGION="us-east-2" # Your AWS region
aws secretsmanager create-secret \
--name "ksk-use2-staging-myapp-db" \
--secret-string "postgres://user:pass@host:5432/db" \
--region ${REGION}
```
Or using the AWS Console:
1. Navigate to Secrets Manager
2. Click "Store a new secret"
3. Choose "Other type of secret"
4. Enter key/value pairs or plaintext
5. Name it following the [naming convention](#secret-naming-conventions)
Create `ExternalSecret.myapp-secrets.yaml` in your service's templates:
```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: myapp-secrets
namespace: myapp
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: myapp-secrets
creationPolicy: Owner
data:
- secretKey: DATABASE_URL
remoteRef:
key: ksk-use2-staging-myapp-db
```
```bash theme={null}
# Apply the ExternalSecret
kubectl apply -f ExternalSecret.myapp-secrets.yaml
# Check the ExternalSecret status
kubectl get externalsecret myapp-secrets -n myapp
# Verify the Secret was created
kubectl get secret myapp-secrets -n myapp
kubectl get secret myapp-secrets -n myapp -o jsonpath='{.data.DATABASE_URL}' | base64 -d
```
## Secret Naming Conventions
The kit uses [CloudPosse null-label](https://github.com/cloudposse/terraform-null-label) for consistent resource naming. Secrets follow the same pattern:
```
{namespace}-{environment}-{stage}-{name}
```
| Component | Description | Examples |
| ------------- | ------------------------- | ------------------------------------------------------- |
| `namespace` | Organization abbreviation | `ksk` (kube-starter-kit), `myco`, etc. |
| `environment` | AWS region abbreviation | `use2` (us-east-2), `use1` (us-east-1), `gbl` (global) |
| `stage` | Deployment stage | `staging`, `prod`, `shared` |
| `name` | Descriptive secret name | `myapp-db`, `argocd-github-dex`, `signoz-ingestion-key` |
The examples below use `ksk` as the namespace (the kit's default). Replace this with the namespace you configured during [Bootstrap Accounts](/usage/getting-started/03-bootstrap-accounts).
Examples:
* `ksk-use2-staging-myapp-db` - Database credentials for myapp
* `ksk-use2-staging-argocd-github-dex` - ArgoCD GitHub OAuth credentials
* `ksk-use2-staging-signoz-ingestion-key` - SigNoz observability ingestion key
This convention:
* Aligns with all other Terraform-managed resources
* Makes secrets easy to identify by cluster/environment
* Enables fine-grained IAM policies using prefixes
* Avoids naming conflicts across environments
## Fetch Multiple Values from One Secret
AWS Secrets Manager secrets can contain JSON with multiple key/value pairs:
```bash theme={null}
export REGION="us-east-2" # Your AWS region
aws secretsmanager create-secret \
--name "ksk-use2-staging-myapp-config" \
--secret-string '{"DB_HOST":"localhost","DB_USER":"admin","DB_PASS":"secret"}' \
--region ${REGION}
```
```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: myapp-config
namespace: myapp
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: myapp-config
data:
- secretKey: DB_HOST
remoteRef:
key: ksk-use2-staging-myapp-config
property: DB_HOST
- secretKey: DB_USER
remoteRef:
key: ksk-use2-staging-myapp-config
property: DB_USER
- secretKey: DB_PASS
remoteRef:
key: ksk-use2-staging-myapp-config
property: DB_PASS
```
## Use Secrets in Pods
Reference the synced Secret in your Deployment:
```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
env:
# Single environment variable
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: myapp-secrets
key: DATABASE_URL
# Or load all keys as env vars
envFrom:
- secretRef:
name: myapp-config
```
## Refresh and Sync Behavior
### Automatic Refresh
ExternalSecrets periodically refresh from AWS based on `refreshInterval`:
```yaml theme={null}
spec:
refreshInterval: 1h # Check for updates every hour
```
### Force Refresh
To immediately sync a secret:
```bash theme={null}
# Annotate to trigger refresh
kubectl annotate externalsecret myapp-secrets -n myapp force-sync=$(date +%s) --overwrite
```
### Reloader Integration
The kit includes [Reloader](https://github.com/stakater/Reloader), which automatically restarts pods when their Secrets change:
```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
annotations:
reloader.stakater.com/auto: "true" # Restart on any Secret/ConfigMap change
```
Or for specific secrets:
```yaml theme={null}
metadata:
annotations:
secret.reloader.stakater.com/reload: "myapp-secrets"
```
## Environment-Specific Secrets
Use different secret paths for each environment:
```yaml theme={null}
# values.yaml (staging)
secrets:
databaseUrlKey: ksk-use2-staging-myapp-db
# values.production.yaml
secrets:
databaseUrlKey: ksk-use2-prod-myapp-db
```
Then template the ExternalSecret:
```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: myapp-secrets
spec:
data:
- secretKey: DATABASE_URL
remoteRef:
key: {{ .Values.secrets.databaseUrlKey }}
```
## IAM Permissions
The External Secrets service account needs permission to read secrets. This is configured via Pod Identity in the EKS Terraform module.
By default, the External Secrets Pod Identity role has access to all secrets (`arn:aws:secretsmanager:*:*:secret:*`). To restrict access to specific prefixes, update the policy in `terraform/modules/eks/base-infra-resources.tf`:
```hcl theme={null}
module "external_secrets_pod_identity" {
# ...
# Replace region and namespace prefix with your values
external_secrets_secrets_manager_arns = [
"arn:aws:secretsmanager::*:secret:--staging-*",
"arn:aws:secretsmanager::*:secret:--prod-*"
]
}
```
## Next Steps
* [Database Operations](/usage/operations/07-database-operations) - Manage database credentials
* [Bootstrapping a New Service](/usage/operations/04-bootstrapping-new-service) - Add secrets for new services
# Database Operations
Source: https://kubestarterkit.com/usage/operations/07-database-operations
Manage CloudNativePG clusters and Atlas migrations
## Overview
The kit uses two components for database management:
* **[CloudNativePG](https://cloudnative-pg.io/)** - PostgreSQL operator for running databases in Kubernetes
* **[Atlas](https://atlasgo.io/)** - Schema migration tool that runs as a Kubernetes Job with Argo CD sync wave ordering
After Planetscale releases their [updated terraform provider](https://planetscale.com/changelog/terraform-provider-v1) (Jan 2026)
I plan to replace CloudNativePG with that as the recommended approach for hosting application databases.
## Architecture
```
┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ Application │ │ Migration Job │ │ CloudNativePG │
│ Deployment │ │ (sync-wave: -1) │ │ Cluster │
└───────────────────┘ └───────────────────┘ └───────────────────┘
│ │ │
│ ▼ │
│ ┌───────────────┐ │
└────────────────>│ PostgreSQL │<────────────────┘
│ Primary │
└───────────────┘
│
┌──────┴──────┐
│ │
┌──────▼────┐ ┌──────▼────┐
│ Replica │ │ Replica │
└───────────┘ └───────────┘
```
Argo CD sync waves ensure the migration Job completes before the application Deployment starts. The migration Job uses `sync-wave: -1` (or lower) to run first.
## CloudNativePG Clusters
### View Existing Clusters
```bash theme={null}
# List all PostgreSQL clusters
kubectl get clusters -A
# Get cluster details
kubectl describe cluster go-backend-cluster -n go-backend
```
### Create a New Cluster
Add a Cluster resource to your service's Kubernetes manifests:
```yaml theme={null}
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: myapp-cluster
namespace: myapp
spec:
instances: 3
storage:
size: 10Gi
storageClass: ebs-gp3-encrypted
postgresql:
parameters:
max_connections: "100"
shared_buffers: "256MB"
bootstrap:
initdb:
database: myapp
owner: myapp
```
### Access the Database
```bash theme={null}
# Get the connection password
kubectl get secret myapp-cluster-app -n myapp -o jsonpath='{.data.password}' | base64 -d
# Port-forward to the primary
kubectl port-forward svc/myapp-cluster-rw -n myapp 5432:5432
# Connect with psql
PGPASSWORD=$(kubectl get secret myapp-cluster-app -n myapp -o jsonpath='{.data.password}' | base64 -d) \
psql -h localhost -U myapp -d myapp
```
### Connection Strings
CloudNativePG creates services for different access patterns:
| Service | Purpose |
| ------------------ | ------------------------- |
| `myapp-cluster-rw` | Read-write (primary only) |
| `myapp-cluster-ro` | Read-only (replicas) |
| `myapp-cluster-r` | Any instance |
Connection string format:
```
postgres://myapp:@myapp-cluster-rw.myapp.svc:5432/myapp
```
## Atlas Migrations
### Migration File Structure
Migrations live in `services/{service}/migrations/`:
```
services/go-backend/
├── migrations/
│ ├── 20251022011957_initial.sql
│ ├── 20251022013319_add_id2.sql
│ └── ...
└── atlas.hcl
```
### Create a New Migration
Create a new file with timestamp prefix:
```bash theme={null}
# Generate timestamp
TIMESTAMP=$(date +%Y%m%d%H%M%S)
# Create migration file
touch services/go-backend/migrations/${TIMESTAMP}_add_email_column.sql
```
Write your migration:
```sql theme={null}
-- Add email column to users table
ALTER TABLE "users" ADD COLUMN "email" character varying;
CREATE INDEX "users_email_idx" ON "users" ("email");
```
Run the migration against your local database:
```bash theme={null}
cd services/go-backend
# Start local postgres
mise run run-postgres
# Apply migrations
atlas migrate apply --env local
```
```bash theme={null}
git add services/go-backend/migrations/
git commit -m "feat: add email column to users"
git push origin main
```
The migration Job runs automatically before the application starts, ordered via Argo CD sync waves.
### Migration Job
Migrations run as a Kubernetes Job before the application starts. Argo CD sync waves ensure proper ordering:
```yaml theme={null}
apiVersion: batch/v1
kind: Job
metadata:
name: myapp-migrations
annotations:
argocd.argoproj.io/sync-wave: "-1" # Run before Deployment (wave 0)
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
template:
spec:
containers:
- name: migrate
image: arigaio/atlas:latest
command: ["atlas", "migrate", "apply", "--env", "kubernetes"]
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: myapp-cluster-app
key: uri
volumeMounts:
- name: migrations
mountPath: /migrations
volumes:
- name: migrations
configMap:
name: myapp-migrations
```
### Atlas Configuration
Configure Atlas in `atlas.hcl`:
```hcl theme={null}
env "local" {
url = "postgres://postgres:password@localhost:5432/myapp?sslmode=disable"
migration {
dir = "file://migrations"
}
}
env "kubernetes" {
url = getenv("DATABASE_URL")
migration {
dir = "file:///migrations"
}
}
```
## Common Operations
### View Migration Status
```bash theme={null}
# Check migration job status
kubectl get jobs -n myapp
# View migration logs
kubectl logs job/myapp-migrations -n myapp
```
### Rollback a Migration
Atlas doesn't support automatic rollbacks. To rollback:
1. Create a new "down" migration that reverses the changes
2. Deploy the rollback migration
```sql theme={null}
-- 20251023120000_rollback_email_column.sql
DROP INDEX IF EXISTS "users_email_idx";
ALTER TABLE "users" DROP COLUMN IF EXISTS "email";
```
### Backup and Restore
CloudNativePG supports continuous backup to S3:
```yaml theme={null}
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: myapp-cluster
spec:
backup:
barmanObjectStore:
destinationPath: s3://myapp-backups/
s3Credentials:
accessKeyId:
name: aws-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: aws-creds
key: SECRET_ACCESS_KEY
wal:
compression: gzip
retentionPolicy: "30d"
```
Trigger a backup:
```bash theme={null}
kubectl apply -f - <│ (DaemonSet) │────>│ │
└───────────────────┘ └───────────────────┘ └───────────────────┘
▲
┌───────────────────┐ │
│ Kubernetes │ │
│ Metrics/Logs │──────────────┘
└───────────────────┘
```
## SigNoz Setup
Sign up at [signoz.io/teams](https://signoz.io/teams/) and create a new project.
In SigNoz Cloud:
1. Go to **Settings → Ingestion Settings**
2. Copy the **Ingestion Key**
3. Note the **Region** (us or eu)
The secret name follows the pattern `{prefix}-{region}-{env}-eks-signoz-apikey`. For example:
```bash theme={null}
export REGION="us-east-2" # Your AWS region
aws secretsmanager create-secret \
--name "ksk-use2-staging-eks-signoz-apikey" \
--secret-string '{"signoz-apikey": "your-ingestion-key"}' \
--region ${REGION}
```
Note the JSON format - the key name `signoz-apikey` must match the `apiKeyExistingSecretKey` value.
The chart uses an umbrella wrapper around the upstream `k8s-infra` chart.
Edit `kubernetes/src/infrastructure/signoz-k8s-infra/values.yaml` (base configuration):
```yaml theme={null}
chartVersions:
k8s-infra: "0.15.0"
k8s-infra:
global:
cloud: aws
otelCollectorEndpoint: ingest.us.signoz.cloud:443
otelInsecure: false
apiKeyExistingSecretName: signoz-apikey
apiKeyExistingSecretKey: signoz-apikey
presets:
otlpExporter:
enabled: true
logsCollection:
enabled: true
```
Then configure environment-specific values in `values.staging.yaml`:
```yaml theme={null}
signozApiKeyAwsSecret: ksk-use2-staging-eks-signoz-apikey
k8s-infra:
global:
clusterName: ksk-use2-staging-eks
deploymentEnvironment: staging
```
The ExternalSecret is already templated at `kubernetes/src/infrastructure/signoz-k8s-infra/templates/ExternalSecret.signoz-apikey.yaml`:
```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: signoz-apikey
namespace: {{.Release.Namespace}}
spec:
refreshInterval: 1h0m0s
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: signoz-apikey
dataFrom:
- extract:
key: {{.Values.signozApiKeyAwsSecret}}
```
This extracts the secret from AWS Secrets Manager using the path defined in `signozApiKeyAwsSecret`.
## Instrument Your Application
### Go Applications
The kit's `go-backend` example includes OpenTelemetry instrumentation:
```go theme={null}
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/trace"
)
func initTracer() (*trace.TracerProvider, error) {
exporter, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint("k8s-infra-otel-agent.signoz.svc:4317"),
otlptracegrpc.WithInsecure(),
)
if err != nil {
return nil, err
}
tp := trace.NewTracerProvider(
trace.WithBatcher(exporter),
trace.WithResource(resource.NewWithAttributes(
semconv.ServiceNameKey.String("go-backend"),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}
```
### Environment Variables
Configure the OTel SDK via environment variables in your Deployment:
```yaml theme={null}
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: "http://k8s-infra-otel-agent.signoz.svc:4317"
- name: OTEL_SERVICE_NAME
value: "myapp"
- name: OTEL_RESOURCE_ATTRIBUTES
value: "service.namespace=myapp,deployment.environment=staging"
```
## View Telemetry in SigNoz
### Traces
1. Open SigNoz Cloud dashboard
2. Navigate to **Traces**
3. Filter by service name or trace ID
4. Click a trace to see the waterfall view
### Metrics
1. Navigate to **Dashboards**
2. Create custom dashboards or use built-in templates
3. Query metrics using PromQL syntax
### Logs
1. Navigate to **Logs**
2. Filter by service, severity, or search text
3. Click a log line to see structured fields
## Kubernetes Metrics
The signoz-k8s-infra collector automatically gathers:
* Node metrics (CPU, memory, disk)
* Pod metrics (resource usage, restarts)
* Container metrics
* Kubernetes events
View in SigNoz under **Infrastructure → Kubernetes**.
## Custom Metrics
Instrument your application with custom metrics:
```go theme={null}
import (
"go.opentelemetry.io/otel/metric"
)
var (
requestCounter metric.Int64Counter
requestLatency metric.Float64Histogram
)
func initMetrics(mp metric.MeterProvider) {
meter := mp.Meter("myapp")
requestCounter, _ = meter.Int64Counter("myapp.requests.total",
metric.WithDescription("Total number of requests"),
)
requestLatency, _ = meter.Float64Histogram("myapp.request.duration",
metric.WithDescription("Request duration in milliseconds"),
metric.WithUnit("ms"),
)
}
func handleRequest(w http.ResponseWriter, r *http.Request) {
start := time.Now()
defer func() {
requestCounter.Add(r.Context(), 1)
requestLatency.Record(r.Context(), float64(time.Since(start).Milliseconds()))
}()
// ... handle request
}
```
## Alerts
Configure alerts in SigNoz Cloud:
1. Go to **Alerts → New Alert**
2. Define the condition (e.g., error rate > 5%)
3. Set severity and notification channels
Supported channels:
* Slack
* PagerDuty
* Email
* Webhooks
## Local Development
For local development with Tilt, signoz-k8s-infra is not deployed by default. You can:
1. **Skip observability locally** - Traces go nowhere, which is fine for development
2. **Run SigNoz locally** (optional):
```bash theme={null}
docker run -d --name signoz \
-p 3301:3301 -p 4317:4317 \
signoz/signoz:latest
```
3. **Configure the collector endpoint**:
```yaml theme={null}
# values.local.yaml
otelCollectorEndpoint: host.docker.internal:4317
```
## Troubleshooting
### No traces appearing in SigNoz
1. Verify the collector is running:
```bash theme={null}
kubectl get pods -n signoz
```
2. Check collector logs:
```bash theme={null}
kubectl logs -n signoz -l app.kubernetes.io/name=otel-collector
```
3. Verify the ingestion key secret exists:
```bash theme={null}
kubectl get secret signoz-apikey -n signoz -o yaml
```
4. Check application OTel configuration:
```bash theme={null}
kubectl logs -n myapp deployment/myapp | grep -i otel
```
### High cardinality warnings
If you see cardinality warnings:
1. Review your metric labels/attributes
2. Avoid high-cardinality values (user IDs, request IDs)
3. Use buckets for continuous values
### Missing Kubernetes metrics
1. Verify the kube-state-metrics pod is running
2. Check RBAC permissions for the collector service account
3. Review collector configuration for kubelet access
## Switching Observability Providers
The kit is designed to work with any OpenTelemetry-compatible backend. To switch from SigNoz:
1. **Disable signoz-k8s-infra** in `kubernetes/src/argocd/infrastructure/values.yaml`
2. **Add your provider's collector** (e.g., Datadog, Grafana Cloud)
3. **Update application OTLP endpoints** to point to the new collector
## Best Practices
1. **Use semantic conventions** - Follow OpenTelemetry naming standards
2. **Set service.name** - Always identify your service in traces
3. **Add context** - Include request IDs, user IDs (hashed) in spans
4. **Sample appropriately** - Use head-based sampling for high-volume services
5. **Alert on SLOs** - Focus alerts on user-impacting metrics