Welcome to from-docker-to-kubernetes

Kubernetes Admission Controllers

Comprehensive guide to Kubernetes Admission Controllers for implementing dynamic security policies and governance controls

Introduction to Admission Controllers

Admission controllers are powerful components in the Kubernetes control plane that intercept and process requests to the API server before object persistence but after authentication and authorization. They serve as critical policy enforcement points:

  • Granular policy enforcement: Apply fine-grained validation and mutation rules to API requests
  • Declarative governance: Define organizational policies in a declarative manner
  • Security guardrails: Prevent misconfigured or non-compliant resources from being created
  • Automated remediation: Dynamically modify resources to conform to policies
  • Audit and compliance: Ensure all created resources meet organizational standards

This guide explores the architecture, implementation, and configuration of admission controllers in Kubernetes, covering both built-in and extensible webhook-based controllers that provide critical security and governance capabilities.

Admission Controller Architecture

The Admission Control Chain

Kubernetes admission control operates as a sequential chain of plugins that process API requests:

  1. Authentication: Establishes the identity of the requestor
  2. Authorization: Determines if the requestor has permission for the operation
  3. Mutating admission: Controllers that can modify the request object
  4. Object Schema Validation: Validates the object against its OpenAPI schema
  5. Validating admission: Controllers that can validate but not modify the request
┌─────────────┐    ┌─────────────┐    ┌────────────────┐    ┌────────────┐    ┌──────────────────┐    ┌──────────┐
│             │    │             │    │                │    │            │    │                  │    │          │
│   Client    │───►│  Authenti-  │───►│  Authori-      │───►│  Mutating  │───►│  Object Schema   │───►│Validating│
│   Request   │    │   cation    │    │   zation       │    │ Admission  │    │   Validation     │    │Admission │
│             │    │             │    │                │    │            │    │                  │    │          │
└─────────────┘    └─────────────┘    └────────────────┘    └────────────┘    └──────────────────┘    └──────────┘
                                                                                                            │
                                                                                                            ▼
                                                                                                      ┌──────────┐
                                                                                                      │          │
                                                                                                      │   etcd   │
                                                                                                      │          │
                                                                                                      └──────────┘

This sequential processing ensures that all policy checks are performed before a resource is persisted to the cluster.

Types of Admission Controllers

Kubernetes implements two primary types of admission controllers:

  1. Built-in admission controllers: Compiled into the API server binary
  2. Dynamic admission controllers: Implemented as webhooks (ValidatingWebhookConfiguration and MutatingWebhookConfiguration)

Each type has specific use cases and configuration approaches:

# View all enabled admission controllers in your cluster
kubectl exec -it kube-apiserver-pod -n kube-system -- kube-apiserver -h | grep "enable-admission-plugins"

Built-in Admission Controllers

Essential Built-in Controllers

Kubernetes includes numerous built-in admission controllers:

Configuring the API Server

To enable or disable built-in admission controllers, configure the kube-apiserver:

# kube-apiserver configuration in a kubeadm cluster
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.25.0
apiServer:
  extraArgs:
    enable-admission-plugins: NodeRestriction,PodSecurity,ResourceQuota,LimitRanger,ServiceAccount,DefaultStorageClass
    disable-admission-plugins: DefaultTolerationSeconds

For non-kubeadm deployments, edit the API server manifest or command line arguments:

# Typical kube-apiserver command line arguments
kube-apiserver \
  --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota,LimitRanger,ServiceAccount,DefaultStorageClass \
  --disable-admission-plugins=DefaultTolerationSeconds \
  # other API server flags...

Dynamic Admission Controllers

Validating Webhooks

Validating admission webhooks can reject requests but cannot modify them:

# ValidatingWebhookConfiguration example
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy-validator
webhooks:
- name: pod-policy.example.com
  clientConfig:
    service:
      namespace: webhook-system
      name: webhook-service
      path: "/validate-pods"
    caBundle: "Ci0tLS0tQk..."
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
    scope: "Namespaced"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  failurePolicy: Fail
  namespaceSelector:
    matchLabels:
      webhook-validation: enabled

Mutating Webhooks

Mutating admission webhooks can modify and validate resources:

# MutatingWebhookConfiguration example
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pod-defaults
webhooks:
- name: defaults.example.com
  clientConfig:
    service:
      namespace: webhook-system
      name: webhook-service
      path: "/mutate-pods"
    caBundle: "Ci0tLS0tQk..."
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE"]
    resources: ["pods"]
    scope: "Namespaced"
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5
  failurePolicy: Ignore
  namespaceSelector:
    matchExpressions:
    - key: environment
      operator: In
      values: ["dev", "staging", "prod"]

Webhook Implementation

Implementing a webhook server requires handling AdmissionReview requests and responses:

// Simplified Go code for a webhook server
package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    
    admission "k8s.io/api/admission/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/serializer"
)

var (
    runtimeScheme = runtime.NewScheme()
    codecs        = serializer.NewCodecFactory(runtimeScheme)
    deserializer  = codecs.UniversalDeserializer()
)

// Validate handles validation requests
func validate(w http.ResponseWriter, r *http.Request) {
    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Failed to read request body", http.StatusBadRequest)
        return
    }
    
    // Parse AdmissionReview
    var admissionReview admission.AdmissionReview
    if _, _, err := deserializer.Decode(body, nil, &admissionReview); err != nil {
        http.Error(w, "Failed to decode request", http.StatusBadRequest)
        return
    }
    
    // Prepare response
    response := admission.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            APIVersion: admission.SchemeGroupVersion.String(),
            Kind:       "AdmissionReview",
        },
    }
    response.Response = &admission.AdmissionResponse{
        UID:     admissionReview.Request.UID,
        Allowed: true,
    }
    
    // Perform validation logic here
    // ...
    
    // Send response
    responseBytes, err := json.Marshal(response)
    if err != nil {
        http.Error(w, "Failed to encode response", http.StatusInternalServerError)
        return
    }
    
    w.Header().Set("Content-Type", "application/json")
    w.Write(responseBytes)
}

func main() {
    http.HandleFunc("/validate-pods", validate)
    fmt.Println("Starting webhook server on port 8443...")
    http.ListenAndServeTLS(":8443", "/certs/tls.crt", "/certs/tls.key", nil)
}

Webhook Deployment

Deploy webhook servers with proper TLS configuration:

# Webhook server deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook-server
  namespace: webhook-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webhook-server
  template:
    metadata:
      labels:
        app: webhook-server
    spec:
      containers:
      - name: server
        image: webhook-server:1.0.0
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: webhook-tls
          mountPath: /certs
          readOnly: true
      volumes:
      - name: webhook-tls
        secret:
          secretName: webhook-server-tls
---
# Service exposing the webhook server
apiVersion: v1
kind: Service
metadata:
  name: webhook-service
  namespace: webhook-system
spec:
  selector:
    app: webhook-server
  ports:
  - port: 443
    targetPort: 8443

Policy Enforcement Examples

PodSecurity Standards

PodSecurity is a built-in admission controller that replaces PodSecurityPolicy:

# Namespace with PodSecurity Standards
apiVersion: v1
kind: Namespace
metadata:
  name: restricted-workloads
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

This configuration enforces the "restricted" policy profile defined in the Kubernetes PodSecurity Standards.

Open Policy Agent (OPA) Integration

OPA Gatekeeper extends Kubernetes with policy enforcement:

# Install OPA Gatekeeper
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.9/deploy/gatekeeper.yaml

# Define a constraint template
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("Missing required labels: %v", [missing])
        }

# Create a constraint instance
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: deploy-must-have-owner
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
  parameters:
    labels: ["owner", "app"]

Kyverno Policies

Kyverno is a policy engine designed specifically for Kubernetes:

# Install Kyverno
kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.8.0/install.yaml

# Create a policy to require resource limits
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: enforce
  rules:
  - name: check-resource-limits
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Resource limits are required for CPU and memory"
      pattern:
        spec:
          containers:
          - name: "*"
            resources:
              limits:
                memory: "?*"
                cpu: "?*"

Advanced Configuration

Failure Policies

Configure webhook behavior when the webhook server is unavailable:

# ValidatingWebhookConfiguration with Fail policy
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: critical-validator
webhooks:
- name: critical.example.com
  # ... other configuration ...
  failurePolicy: Fail  # Request is rejected if webhook is unavailable
# MutatingWebhookConfiguration with Ignore policy
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: optional-mutator
webhooks:
- name: optional.example.com
  # ... other configuration ...
  failurePolicy: Ignore  # Request proceeds if webhook is unavailable

Reinvocation Policy

Control whether mutating webhooks are called again after object mutations:

# MutatingWebhookConfiguration with reinvocation policy
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: multi-stage-mutator
webhooks:
- name: stage1.example.com
  # ... other configuration ...
  reinvocationPolicy: IfNeeded  # Call again if another webhook mutates the object

Timeout Configuration

Set appropriate timeouts to balance reliability and performance:

# Webhook with timeout configuration
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: performance-validator
webhooks:
- name: performance.example.com
  # ... other configuration ...
  timeoutSeconds: 3  # Short timeout for performance-critical path

Best Practices

Security Considerations

Implement these security practices for admission controllers:

  1. TLS configuration: Use strong TLS settings for webhook servers
  2. Certificate rotation: Automate certificate renewal for webhooks
  3. Least privilege: Limit webhook permissions to required resources
  4. Namespace isolation: Run webhook servers in dedicated namespaces
  5. Resource limits: Set appropriate CPU and memory limits for webhook pods

Performance Optimization

Optimize admission controllers for performance:

  1. Selective targeting: Use precise resource selectors to minimize webhook calls
  2. Efficient processing: Optimize webhook code for fast response times
  3. Appropriate timeouts: Set realistic timeouts based on webhook complexity
  4. Caching: Implement caching for expensive validation operations
  5. Graduated rollout: Deploy policies to non-critical namespaces first

Progressive Implementation

Implement admission controllers with a phased approach:

  1. Audit mode: Start with warning-only enforcement
  2. Limited scope: Apply to specific namespaces before cluster-wide rollout
  3. Incremental policies: Add policies one at a time, not all at once
  4. Exemption mechanisms: Create override mechanisms for emergency situations
  5. User education: Provide clear documentation and guidance for users

Troubleshooting

Debugging Webhook Issues

Troubleshoot common webhook problems:

  1. Check webhook logs:
    kubectl logs -n webhook-system -l app=webhook-server
    
  2. Verify webhook configuration:
    kubectl get validatingwebhookconfiguration
    kubectl get mutatingwebhookconfiguration
    
  3. Inspect webhook configuration details:
    kubectl describe validatingwebhookconfiguration pod-policy-validator
    
  4. Check API server logs for webhook errors:
    kubectl logs -n kube-system -l component=kube-apiserver
    
  5. Test webhook connectivity:
    kubectl run webhook-test --image=curlimages/curl -i --rm --restart=Never -- \
      curl -k https://webhook-service.webhook-system.svc:443/health
    

Common Failure Modes

Address these common webhook failure scenarios:

  1. TLS certificate issues: Ensure CA bundle in webhook configuration matches server certificate
  2. Network connectivity: Verify network policies allow API server to webhook communication
  3. Webhook timeouts: Increase timeoutSeconds if webhooks perform complex operations
  4. Resource constraints: Add appropriate resource requests and limits to webhook deployments
  5. Logic errors: Use logging and monitoring to identify issues in webhook logic

Conclusion

Kubernetes admission controllers provide powerful mechanisms for implementing security policies, governance controls, and automation across your clusters. By leveraging both built-in controllers and dynamic webhooks, you can ensure that all resources conform to organizational standards before they're created or modified.

As your Kubernetes environments grow in scale and complexity, admission controllers become increasingly important for maintaining security posture and operational consistency. The flexibility of the webhook system allows for virtually unlimited customization while preserving the declarative nature of Kubernetes resource management.

Remember to implement admission controls gradually, with appropriate testing and escape hatches, to avoid disrupting critical workloads. With careful planning and implementation, admission controllers can significantly enhance your cluster's security and compliance posture.