ConfigMaps & Secrets

Configuration Management

Kubernetes provides ConfigMaps and Secrets to decouple configuration from pod specifications. This separation of concerns allows you to change application configuration without rebuilding container images, making your applications more portable and easier to manage across different environments (development, staging, production).

ConfigMaps

ConfigMaps store non-confidential configuration data in key-value pairs. They can contain entire configuration files or individual properties, and are suitable for environment-specific settings, configuration files, command-line arguments, and other non-sensitive configuration data.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: default # Optional: Defaults to "default" if not specified
  labels: # Optional: Add labels for organization
    app: myapp
    environment: production
data:
  # You can store entire configuration files
  app.properties: |
    color.background=blue
    color.foreground=white
    app.mode=production
    retry.count=3
  # Different file formats are supported
  app.yml: |
    server:
      port: 8080
      timeout: 30s
    logging:
      level: info
  # Or individual configuration values
  database_url: mongodb://localhost:27017
  cache_ttl: "300"

# From literal values
kubectl create configmap app-config \
  --from-literal=database_url=mongodb://localhost:27017 \
  --from-literal=cache_ttl=300

# From files
kubectl create configmap app-config \
  --from-file=app.properties \
  --from-file=app.yml

# From directory (all files in the directory)
kubectl create configmap app-config --from-file=config-dir/

apiVersion: v1
kind: Pod
metadata:
  name: config-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    # 1. Load all values from ConfigMap as environment variables
    envFrom:
    - configMapRef:
        name: app-config
        # Optional: prefix all environment variables
        prefix: CONFIG_
    # 2. Mount ConfigMap as a volume (creates files for each key)
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
      readOnly: true # Best practice: mount as read-only
    # 3. Optional: Use ConfigMap values in command/args
    command: ["java"]
    args: ["-Dserver.port=$(SERVER_PORT)", "-jar", "app.jar"]
    env:
    # 4. Selectively use specific keys from ConfigMap
    - name: SERVER_PORT
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: server.port
          # Optional: make this reference optional
          optional: true
  volumes:
  - name: config-volume
    configMap:
      name: app-config
      # Optional: include only specific items
      items:
      - key: app.properties
        path: application.properties
      - key: app.yml
        path: config.yml
      # Optional: set file permissions
      defaultMode: 0400 # Read-only by owner

Secrets

Secrets are similar to ConfigMaps but are specifically designed for storing sensitive information such as passwords, OAuth tokens, and SSH keys. Kubernetes provides additional security measures for Secrets, although they should be used in conjunction with other security controls for production environments.

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: default
  labels:
    app: myapp
type: Opaque  # Generic secret type
data:
  # Values must be base64 encoded
  username: YWRtaW4=          # "admin" in base64
  password: cGFzc3dvcmQ=      # "password" in base64
  api_token: dG9rZW4xMjM0NTY=  # "token123456" in base64
# Alternatively, use stringData for unencoded values (Kubernetes will encode them)
stringData:
  config.json: |
    {
      "apiKey": "xyz123",
      "environment": "production"
    }

# From literal values (automatically encoded)
kubectl create secret generic app-secrets \
  --from-literal=username=admin \
  --from-literal=password=password

# From files (automatically encoded)
kubectl create secret generic tls-certs \
  --from-file=cert.pem \
  --from-file=key.pem

# Create TLS secret directly from cert/key files
kubectl create secret tls tls-secret \
  --cert=cert.pem \
  --key=key.pem

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    # 1. As individual environment variables
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: username
          optional: false # Default is false - will fail if secret/key not found
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: password
    
    # 2. All keys as environment variables
    envFrom:
    - secretRef:
        name: app-secrets
        
    # 3. Mount as volume (creates a file for each key)
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true # Always use readOnly for secrets
  
  volumes:
  - name: secret-volume
    secret:
      secretName: app-secrets
      # Optional: include only specific items
      items:
      - key: config.json
        path: app-config.json
      # Set restrictive permissions
      defaultMode: 0400 # Read-only by owner
  
  # Optional: specify which service account to use
  serviceAccountName: restricted-sa

Configuration Best Practices

Environment Variables

Environment variables are the most common way to pass configuration to containers in Kubernetes, as most applications are already designed to read configuration from environment variables.

From ConfigMap

spec:
  containers:
  - name: app
    image: myapp:1.0
    env:
    - name: APP_COLOR  # Name of the environment variable in the container
      valueFrom:
        configMapKeyRef:
          name: app-config  # Name of the ConfigMap
          key: color.background  # Key in the ConfigMap
          optional: false  # Whether this reference is optional
    
    # You can mix environment variables from different sources
    - name: NODE_ENV
      value: "production"  # Hardcoded value
    
    # Reference values from the pod's field
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    
    # Reference resource limits
    - name: CPU_LIMIT
      valueFrom:
        resourceFieldRef:
          containerName: app
          resource: limits.cpu

From Multiple Sources

You can inject all values from multiple ConfigMaps and Secrets as environment variables at once, which is useful for applications that expect a large number of configuration parameters.

spec:
  containers:
  - name: app
    image: myapp:1.0
    # All keys from these sources become environment variables
    envFrom:
    - configMapRef:
        name: app-config
        optional: false  # Default is false - will fail if ConfigMap not found
    - configMapRef:
        name: common-config
        # Optionally add a prefix to all variables from this source
        prefix: COMMON_
    - secretRef:
        name: app-secrets
    - secretRef:
        name: database-credentials
        prefix: DB_
    
    # Environment variables are merged from all sources
    # If conflicts occur, later sources take precedence

Volume Mounts

Mounting ConfigMaps and Secrets as volumes creates files inside your container, with each key becoming a file and its value becoming the file content. This is ideal for applications that expect to read configuration from files.

spec:
  containers:
  - name: app
    volumeMounts:
    - name: config
      mountPath: /etc/config  # Directory where files will be created
      readOnly: true  # Best practice: mount as read-only
  volumes:
  - name: config
    configMap:
      name: app-config
      # Optional: Select specific items to mount
      items:
      - key: app.properties  # Key in the ConfigMap
        path: application.properties  # Filename to create
      - key: app.yml
        path: config/app.yml  # Can include subdirectories
      # Set permissions for all files
      defaultMode: 0644  # Read-write for owner, read-only for others

spec:
  containers:
  - name: app
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true  # Always use readOnly for secrets
  volumes:
  - name: secrets
    secret:
      secretName: app-secrets
      # Optional: select specific keys
      items:
      - key: tls.crt
        path: certs/tls.crt
      - key: tls.key
        path: private/tls.key
      # Set restrictive permissions
      defaultMode: 0400  # Read-only by owner

Dynamic Updates

Understanding how configuration updates propagate to running containers is crucial for managing applications in Kubernetes.

Usage considerations:

• For configuration that changes frequently, use volume mounts
• For configuration that rarely changes, environment variables are simpler
• For critical configuration, consider immutable ConfigMaps with explicit version labeling
• Applications should handle configuration reloads gracefully

Secret Types

Kubernetes supports several built-in Secret types for common use cases, each with specific expected data fields.

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    my-company.com/secret-rotation: "30d"  # Custom annotations for management
type: mycompany.com/secret-type  # Use a domain-prefixed type for custom types
data:
  custom-key: dmFsdWU=  # base64 encoded value
  api-token: ZXhhbXBsZS10b2tlbg==  # base64 encoded "example-token"

Configuration Management Tools

As applications grow in complexity, manual management of ConfigMaps and Secrets becomes cumbersome. Kubernetes ecosystem offers several tools to manage configuration at scale.

Security Considerations

Configuration and secret management require careful security considerations to prevent data exposure.

Troubleshooting

Configuration-related issues are common in Kubernetes deployments. Here are the most frequent problems and their solutions:

Common issues and solutions:

1. ConfigMap not updating
   • Symptoms: Changes to ConfigMap not reflected in application
   • Causes:
     • Environment variables don't update automatically
     • Volume propagation delay (can take up to 60 seconds)
     • Application not reloading configuration
   • Solutions:
     • Restart pods for environment variable updates
     • Add a configuration reload mechanism to your application
     • Use a sidecar container to watch for changes and signal the application
     • Check if ConfigMap is marked as immutable
2. Secret mounting issues
   • Symptoms: Application can't access secrets, missing files
   • Causes:
     • Secret doesn't exist in the namespace
     • Secret key doesn't match expected name
     • Incorrect mount path
   • Solutions:
     • Verify Secret exists: kubectl get secret <name> -n <namespace>
     • Check pod events: kubectl describe pod <pod-name>
     • Inspect volume mounts: kubectl describe pod <pod-name> | grep -A 10 Mounts
     • Debug with a temporary pod: kubectl run debug --image=busybox -- sleep 3600
3. Permission problems
   • Symptoms: Forbidden errors, RBAC denials
   • Causes:
     • Insufficient RBAC permissions
     • Service account doesn't have access
     • Pod security policies blocking access
   • Solutions:
     • Check pod service account: kubectl get pod <pod-name> -o jsonpath='{.spec.serviceAccountName}'
     • Verify permissions: kubectl auth can-i --as=system:serviceaccount:<namespace>:<sa-name> get secrets
     • Review audit logs for RBAC failures
     • Update RBAC roles and bindings as needed
4. Encoding/decoding errors
   • Symptoms: Garbled configuration data, application errors
   • Causes:
     • Improper base64 encoding in Secrets
     • Non-UTF8 characters in configuration
     • Line ending issues
   • Solutions:
     • Ensure proper base64 encoding: echo -n 'my-value' | base64
     • Use the stringData field in Secrets to avoid manual encoding
     • Check encoding with: kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 --decode
     • Use kubectl create secret commands instead of manual YAML
5. Volume mount failures
   • Symptoms: Pod stuck in init or won't start
   • Causes:
     • ConfigMap or Secret doesn't exist
     • Volume path conflicts
     • Permission issues on the mounted directory
   • Solutions:
     • Check pod events: kubectl describe pod <pod-name>
     • Verify the ConfigMap/Secret exists before the pod starts
     • Add optional: true for non-critical configurations
     • Fix path conflicts or use subdirectories
6. Too large ConfigMaps or Secrets
   • Symptoms: API server rejection, pod startup failure
   • Causes:
     • Kubernetes limits size of ConfigMaps/Secrets (typically 1MB)
     • Too many keys or large values
   • Solutions:
     • Split large configurations into multiple objects
     • Store large files externally (S3, Git, etc.)
     • Use init containers to fetch configuration at startup
     • Consider ConfigMap/Secret projection with specific keys

Advanced Usage

Immutable ConfigMaps

Kubernetes 1.19+ supports immutable ConfigMaps and Secrets, which prevents accidental modifications and improves performance by reducing watches.

apiVersion: v1
kind: ConfigMap
metadata:
  name: immutable-config
  namespace: default
  labels:
    version: "v1.2.3"  # Good practice to version immutable configs
immutable: true  # Prevents any further updates to this ConfigMap
data:
  app.properties: |
    key=value
    environment=production
    feature.new=true

Benefits of immutable configurations:

• Prevent accidental changes
• Improve performance (kube-apiserver load reduction)
• Simplify auditing and compliance
• Enable easier tracking of which configuration version is deployed

Secret Data Keys

Different Secret types require specific keys. For TLS Secrets:

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret
  annotations:
    cert-manager.io/certificate-name: "example-com"  # Optional annotations for management
type: kubernetes.io/tls
data:
  # Required keys for TLS type
  tls.crt: base64-encoded-cert  # Full certificate chain
  tls.key: base64-encoded-key   # Private key
  # Optional additional keys
  ca.crt: base64-encoded-ca     # CA certificate for validation

Init Containers

For complex configuration scenarios, init containers can be used to prepare or transform configuration before the main container starts.

apiVersion: v1
kind: Pod
metadata:
  name: config-init-pod
spec:
  initContainers:
  - name: config-init
    image: busybox
    command: ['sh', '-c', 'wget -O /work-dir/config.json http://config-service/generate && chmod 644 /work-dir/config.json']
    volumeMounts:
    - name: workdir
      mountPath: /work-dir
  containers:
  - name: app
    image: myapp:1.0
    volumeMounts:
    - name: workdir
      mountPath: /etc/app/config
  volumes:
  - name: workdir
    emptyDir: {}

Init containers can:

• Download configuration from external sources
• Merge multiple configuration sources
• Transform configuration into application-specific formats
• Generate dynamic configuration values (e.g., passwords)
• Validate configuration before application starts

Best Practices Checklist