Observability

​

Overview

​

Architecture

┌───────────────────┐     ┌───────────────────┐     ┌───────────────────┐
│   Application     │     │   OTel Collector  │     │   SigNoz Cloud    │
│   (instrumented)  │────>│   (DaemonSet)     │────>│                   │
└───────────────────┘     └───────────────────┘     └───────────────────┘
                                   ▲
┌───────────────────┐              │
│   Kubernetes      │              │
│   Metrics/Logs    │──────────────┘
└───────────────────┘

​

SigNoz Setup

export REGION="us-east-2"  # Your AWS region

aws secretsmanager create-secret \
  --name "ksk-use2-staging-eks-signoz-apikey" \
  --secret-string '{"signoz-apikey": "your-ingestion-key"}' \
  --region ${REGION}

chartVersions:
  k8s-infra: "0.15.0"

k8s-infra:
  global:
    cloud: aws
  otelCollectorEndpoint: ingest.us.signoz.cloud:443
  otelInsecure: false
  apiKeyExistingSecretName: signoz-apikey
  apiKeyExistingSecretKey: signoz-apikey
  presets:
    otlpExporter:
      enabled: true
    logsCollection:
      enabled: true

signozApiKeyAwsSecret: ksk-use2-staging-eks-signoz-apikey
k8s-infra:
  global:
    clusterName: ksk-use2-staging-eks
    deploymentEnvironment: staging

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: signoz-apikey
  namespace: {{.Release.Namespace}}
spec:
  refreshInterval: 1h0m0s
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: signoz-apikey
  dataFrom:
    - extract:
        key: {{.Values.signozApiKeyAwsSecret}}

​

Instrument Your Application

​

Go Applications

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(ctx,
        otlptracegrpc.WithEndpoint("k8s-infra-otel-agent.signoz.svc:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.ServiceNameKey.String("go-backend"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

​

Environment Variables

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://k8s-infra-otel-agent.signoz.svc:4317"
  - name: OTEL_SERVICE_NAME
    value: "myapp"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.namespace=myapp,deployment.environment=staging"

​

View Telemetry in SigNoz

​

Traces

1. Open SigNoz Cloud dashboard
2. Navigate to Traces
3. Filter by service name or trace ID
4. Click a trace to see the waterfall view

​

Metrics

1. Navigate to Dashboards
2. Create custom dashboards or use built-in templates
3. Query metrics using PromQL syntax

​

Logs

1. Navigate to Logs
2. Filter by service, severity, or search text
3. Click a log line to see structured fields

​

Kubernetes Metrics

• Node metrics (CPU, memory, disk)
• Pod metrics (resource usage, restarts)
• Container metrics
• Kubernetes events

​

Custom Metrics

import (
    "go.opentelemetry.io/otel/metric"
)

var (
    requestCounter metric.Int64Counter
    requestLatency metric.Float64Histogram
)

func initMetrics(mp metric.MeterProvider) {
    meter := mp.Meter("myapp")
    
    requestCounter, _ = meter.Int64Counter("myapp.requests.total",
        metric.WithDescription("Total number of requests"),
    )
    
    requestLatency, _ = meter.Float64Histogram("myapp.request.duration",
        metric.WithDescription("Request duration in milliseconds"),
        metric.WithUnit("ms"),
    )
}

func handleRequest(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    defer func() {
        requestCounter.Add(r.Context(), 1)
        requestLatency.Record(r.Context(), float64(time.Since(start).Milliseconds()))
    }()
    // ... handle request
}

​

Alerts

​

Local Development

1. Skip observability locally - Traces go nowhere, which is fine for development
2. Run SigNoz locally (optional):
   Copy

   docker run -d --name signoz \
     -p 3301:3301 -p 4317:4317 \
     signoz/signoz:latest
   
3. Configure the collector endpoint:
   Copy

   # values.local.yaml
   otelCollectorEndpoint: host.docker.internal:4317
   

​

Troubleshooting

​

No traces appearing in SigNoz

1. Verify the collector is running:
   Copy

   kubectl get pods -n signoz
   
2. Check collector logs:
   Copy

   kubectl logs -n signoz -l app.kubernetes.io/name=otel-collector
   
3. Verify the ingestion key secret exists:
   Copy

   kubectl get secret signoz-apikey -n signoz -o yaml
   
4. Check application OTel configuration:
   Copy

   kubectl logs -n myapp deployment/myapp | grep -i otel
   

​

High cardinality warnings

1. Review your metric labels/attributes
2. Avoid high-cardinality values (user IDs, request IDs)
3. Use buckets for continuous values

​

Missing Kubernetes metrics

1. Verify the kube-state-metrics pod is running
2. Check RBAC permissions for the collector service account
3. Review collector configuration for kubelet access

​

Switching Observability Providers

1. Disable signoz-k8s-infra in kubernetes/src/argocd/infrastructure/values.yaml
2. Add your provider’s collector (e.g., Datadog, Grafana Cloud)
3. Update application OTLP endpoints to point to the new collector

​

Best Practices

1. Use semantic conventions - Follow OpenTelemetry naming standards
2. Set service.name - Always identify your service in traces
3. Add context - Include request IDs, user IDs (hashed) in spans
4. Sample appropriately - Use head-based sampling for high-volume services
5. Alert on SLOs - Focus alerts on user-impacting metrics