Welcome to from-docker-to-kubernetes

Security

Understanding Kubernetes security concepts, best practices, and implementations

Kubernetes Security

Kubernetes security is a multi-layered approach that includes cluster security, application security, and network security. Securing a Kubernetes environment requires understanding various security aspects, from infrastructure to application code.

Security in Kubernetes involves protecting the entire container lifecycle, from build phase (securing container images) to runtime security (what containers can do and access during execution) to the cluster infrastructure itself. Each layer requires specific security controls and practices.

Authentication & Authorization

Authentication verifies who can access the cluster, while authorization determines what they can do.

Authentication Methods

  • X.509 Certificates: PKI-based client certificates for strong authentication
  • Service Accounts: Kubernetes-managed identities for pods and applications
  • OpenID Connect: Integration with external identity providers (like OAuth)
  • Webhook Token Auth: Custom authentication logic via webhooks
  • Static Token File: Simple file-based token authentication (not recommended for production)

Role-Based Access Control (RBAC)

RBAC provides fine-grained access policies for Kubernetes resources. A Role defines permissions within a specific namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]  # "" indicates the core API group
  resources: ["pods"]  # The resource type to control access to
  verbs: ["get", "watch", "list"]  # Allowed operations on the resource

Role Bindings

RoleBindings assign Roles to users, groups, or service accounts within a namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: User  # Can be User, Group, or ServiceAccount
  name: jane  # The name of the user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role  # Can be Role or ClusterRole
  name: pod-reader  # Must match the name of the Role
  apiGroup: rbac.authorization.k8s.io

For cluster-wide permissions, use ClusterRoles and ClusterRoleBindings instead.

Pod Security

Pod security focuses on constraining what pods can do during runtime.

Security Context

Security contexts define privilege and access control settings for pods and containers.

apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000  # User ID to run all containers processes
    runAsGroup: 3000  # Group ID for all containers
    fsGroup: 2000  # Group ID for volume access
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    securityContext:
      allowPrivilegeEscalation: false  # Prevents escalating privileges
      capabilities:
        drop:
          - ALL  # Removes all Linux capabilities
      readOnlyRootFilesystem: true  # Optional: Makes root filesystem read-only
      runAsNonRoot: true  # Optional: Validates container doesn't run as root

Pod-level security context settings apply to all containers, while container-level settings override pod-level settings for that specific container.

Pod Security Standards

Pod Security Standards

Pod Security Standards define three different security levels:

  1. Privileged: Unrestricted policy with maximum permissions
  2. Baseline: Minimally restrictive policy preventing known privilege escalations
  3. Restricted: Heavily restricted policy following security best practices

Pod Security Admission

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Configuration Areas

  • Privilege controls: Preventing privilege escalation, root access
  • Capability management: Controlling Linux capabilities
  • Volume restrictions: Limiting accessible volume types
  • Host namespaces: Controlling access to host network/PID/IPC
  • User/group controls: Defining which users containers run as
  • Seccomp/AppArmor/SELinux: System-level security profiles

Network Security

By default, all pods can communicate with all other pods in the cluster. Network policies restrict this communication.

Network Policies

Network policies act as firewalls, controlling pod-to-pod and external communications.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
spec:
  podSelector:  # This policy applies to pods with app=api
    matchLabels:
      app: api
  policyTypes:  # Policy controls both incoming and outgoing traffic
  - Ingress
  - Egress
  ingress:  # Incoming traffic rules
  - from:
    - podSelector:  # Allow traffic only from frontend pods
        matchLabels:
          role: frontend
    - namespaceSelector:  # Optional: Allow from specific namespaces
        matchLabels:
          purpose: production
    ports:  # Restrict to specific port
    - protocol: TCP
      port: 8080
  egress:  # Optional: Control outgoing traffic
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

TLS Configuration

  • Certificate management: Securely managing and rotating TLS certificates
  • Ingress TLS: Encrypting external traffic to applications
  • API server security: Secure communication with the Kubernetes API
  • etcd encryption: Protecting sensitive data in the cluster's datastore
  • mTLS with service mesh: Mutual TLS authentication between services

Secrets Management

Kubernetes Secrets store and manage sensitive information like passwords, tokens, and keys.

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque  # Generic secret type
data:
  # Values must be base64-encoded
  username: YWRtaW4=          # "admin" in base64
  password: MWYyZDFlMmU2N2Rm  # base64-encoded password

You can create secrets using:

# From literal values
kubectl create secret generic mysecret --from-literal=username=admin --from-literal=password=pass123

# From files
kubectl create secret generic ssl-cert --from-file=cert.pem --from-file=key.pem

Security Best Practices

Cluster Hardening

Hardening your Kubernetes cluster involves securing all components from the infrastructure to the application.

Control Plane Security

  • API server security: Restrict access with proper authentication, RBAC, and admission controllers
  • etcd encryption: Enable encryption at rest for sensitive data in etcd
  • Controller manager: Configure controllers securely with appropriate permissions
  • Scheduler security: Protect the scheduler from unauthorized access and manipulation
  • Communication security: Encrypt all control plane communications with TLS

Node Security

  • Kubelet configuration: Secure kubelet authentication, authorization, and anonymous access
  • Container runtime: Use secure container runtime configurations (e.g., gVisor, kata containers)
  • Node access control: Minimize SSH access, use bastion hosts, implement proper user management
  • System hardening: Follow CIS benchmarks for OS hardening, reduce attack surface
  • Node isolation: Implement proper network segmentation for node-to-node communication

Container Security

  • Image scanning: Continuously scan for vulnerabilities in container images
  • Runtime security: Implement runtime security controls (seccomp, AppArmor, SELinux)
  • Resource isolation: Set proper resource limits and requests to prevent DoS
  • Vulnerability management: Process for tracking and remediating CVEs
  • Build security: Secure CI/CD pipelines, sign images, use minimal base images

Audit Logging

Audit logging records actions performed in your Kubernetes cluster, essential for security monitoring and compliance.

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log pod changes at RequestResponse level (includes request and response bodies)
- level: RequestResponse
  resources:
  - group: ""  # Core API group
    resources: ["pods"]
  
# Log ConfigMap and Secret read/write operations  
- level: Metadata
  resources:
  - group: "" 
    resources: ["configmaps", "secrets"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]
    
# Log all other resources at the Metadata level
- level: Metadata
  resources:
  - group: "" 
    resources: ["*"]

Audit policy levels from least to most verbose:

  1. None: Don't log events
  2. Metadata: Log request metadata (user, timestamp, resource, verb) but not request or response body
  3. Request: Log event metadata and request body but not response body
  4. RequestResponse: Log event metadata, request and response bodies

Configure the kube-apiserver with:

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=30  # Optional: retention days

Compliance and Standards

Kubernetes environments may need to comply with various security standards and regulations.

Security Monitoring

Continuous monitoring is essential for maintaining the security posture of your Kubernetes environment.

Tools and Practices

  • Security scanners: Regular scanning of images and infrastructure (Trivy, Clair, Anchore)
  • Log analysis: Centralized logging with anomaly detection (ELK, Splunk, Datadog)
  • Intrusion detection: Runtime security monitoring (Falco, Sysdig Secure, Aqua)
  • Vulnerability scanning: Continuous vulnerability assessment (Nessus, Qualys, OpenVAS)
  • Compliance monitoring: Automated compliance checking (Kube-bench, OPA Gatekeeper)
  • Admission controllers: Policy enforcement at admission time (PodSecurityPolicy, OPA)
  • Runtime protection: Detecting and preventing suspicious behaviors

Response Plan

  • Incident detection: Automated alerting based on security events and anomalies
  • Response procedures: Documented steps for handling different types of security incidents
    • Container compromise
    • Unauthorized access
    • Data exposure
    • Denial of service
  • Recovery process: Methods for secure recovery after an incident
    • Isolation procedures
    • Evidence collection
    • Clean-up and restoration
  • Documentation: Thorough documentation of incidents, responses, and lessons learned
  • Team responsibilities: Clear roles and responsibilities during security incidents
    • Incident commander
    • Technical lead
    • Communications lead
    • Executive sponsor

Advanced Security Topics

Runtime Security

Runtime security focuses on protecting containers during execution. OpenShift's SecurityContextConstraints provide an example of advanced runtime controls:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: restricted
allowPrivilegeEscalation: false  # Prevents privilege escalation
runAsUser:
  type: MustRunAsRange  # User ID must be in a specific range
seLinuxContext:
  type: MustRunAs  # Must use specified SELinux context
allowedCapabilities: []  # No additional capabilities allowed
fsGroup:
  type: MustRunAs  # File system group must be in specific range
readOnlyRootFilesystem: true  # Root filesystem is read-only
volumes:  # Restricts volume types
- configMap
- downwardAPI
- emptyDir
- persistentVolumeClaim
- projected
- secret

For vanilla Kubernetes, similar controls can be implemented using Pod Security Standards and admission controllers like OPA Gatekeeper or Kyverno.

Encryption Configuration

Kubernetes can encrypt Secret resources and other sensitive data at rest in etcd:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets  # Resources to encrypt
    - configmaps  # Optional: encrypt additional resources
    providers:  # List of encryption providers in order of preference
    - aescbc:  # AES-CBC encryption with PKCS#7 padding
        keys:
        - name: key1
          secret: <base64-encoded-key>  # 32-byte key encoded in base64
    - secretbox:  # Optional: Alternative encryption using XSalsa20 and Poly1305
        keys:
        - name: key1
          secret: <base64-encoded-key>
    - identity: {}  # Optional fallback that doesn't encrypt

Apply this configuration to the kube-apiserver with:

--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

After enabling encryption, existing secrets can be encrypted with:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Security Checklist