Security
Understanding Kubernetes security concepts, best practices, and implementations
Kubernetes Security
Kubernetes security is a multi-layered approach that includes cluster security, application security, and network security. Securing a Kubernetes environment requires understanding various security aspects, from infrastructure to application code.
Security in Kubernetes involves protecting the entire container lifecycle, from build phase (securing container images) to runtime security (what containers can do and access during execution) to the cluster infrastructure itself. Each layer requires specific security controls and practices.
Authentication & Authorization
Authentication verifies who can access the cluster, while authorization determines what they can do.
Authentication Methods
- X.509 Certificates: PKI-based client certificates for strong authentication
- Service Accounts: Kubernetes-managed identities for pods and applications
- OpenID Connect: Integration with external identity providers (like OAuth)
- Webhook Token Auth: Custom authentication logic via webhooks
- Static Token File: Simple file-based token authentication (not recommended for production)
Role-Based Access Control (RBAC)
RBAC provides fine-grained access policies for Kubernetes resources. A Role defines permissions within a specific namespace.
Role Bindings
RoleBindings assign Roles to users, groups, or service accounts within a namespace.
For cluster-wide permissions, use ClusterRoles and ClusterRoleBindings instead.
Pod Security
Pod security focuses on constraining what pods can do during runtime.
Security Context
Security contexts define privilege and access control settings for pods and containers.
Pod-level security context settings apply to all containers, while container-level settings override pod-level settings for that specific container.
Pod Security Standards
Note: Pod Security Policies (PSP) were deprecated in Kubernetes v1.21 and removed in v1.25. The recommended replacement is Pod Security Standards implemented via Pod Security Admission Controller.
Pod Security Standards
Pod Security Standards define three different security levels:
- Privileged: Unrestricted policy with maximum permissions
- Baseline: Minimally restrictive policy preventing known privilege escalations
- Restricted: Heavily restricted policy following security best practices
Pod Security Admission
Configuration Areas
- Privilege controls: Preventing privilege escalation, root access
- Capability management: Controlling Linux capabilities
- Volume restrictions: Limiting accessible volume types
- Host namespaces: Controlling access to host network/PID/IPC
- User/group controls: Defining which users containers run as
- Seccomp/AppArmor/SELinux: System-level security profiles
Network Security
By default, all pods can communicate with all other pods in the cluster. Network policies restrict this communication.
Network Policies
Network policies act as firewalls, controlling pod-to-pod and external communications.
TLS Configuration
- Certificate management: Securely managing and rotating TLS certificates
- Ingress TLS: Encrypting external traffic to applications
- API server security: Secure communication with the Kubernetes API
- etcd encryption: Protecting sensitive data in the cluster's datastore
- mTLS with service mesh: Mutual TLS authentication between services
Secrets Management
Kubernetes Secrets store and manage sensitive information like passwords, tokens, and keys.
You can create secrets using:
By default, Kubernetes secrets are stored unencrypted in etcd. For production environments, enable etcd encryption or use external secret management solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.
Security Best Practices
- Use namespace isolation: Logically separate workloads with different security requirements
- Implement RBAC properly: Follow principle of least privilege for all users and service accounts
- Enable audit logging: Track all actions performed in the cluster for security analysis
- Regular security updates: Keep Kubernetes and all components up to date with security patches
- Use network policies: Implement "zero trust" networking by explicitly defining allowed communications
- Encrypt sensitive data: Enable etcd encryption and use proper secret management
- Implement pod security: Use Pod Security Standards to enforce security policies
- Monitor cluster security: Implement continuous security monitoring and scanning
- Scan container images: Use tools to detect vulnerabilities in container images
- Secure CI/CD pipeline: Implement security controls throughout your deployment pipeline
Cluster Hardening
Hardening your Kubernetes cluster involves securing all components from the infrastructure to the application.
Control Plane Security
- API server security: Restrict access with proper authentication, RBAC, and admission controllers
- etcd encryption: Enable encryption at rest for sensitive data in etcd
- Controller manager: Configure controllers securely with appropriate permissions
- Scheduler security: Protect the scheduler from unauthorized access and manipulation
- Communication security: Encrypt all control plane communications with TLS
Node Security
- Kubelet configuration: Secure kubelet authentication, authorization, and anonymous access
- Container runtime: Use secure container runtime configurations (e.g., gVisor, kata containers)
- Node access control: Minimize SSH access, use bastion hosts, implement proper user management
- System hardening: Follow CIS benchmarks for OS hardening, reduce attack surface
- Node isolation: Implement proper network segmentation for node-to-node communication
Container Security
- Image scanning: Continuously scan for vulnerabilities in container images
- Runtime security: Implement runtime security controls (seccomp, AppArmor, SELinux)
- Resource isolation: Set proper resource limits and requests to prevent DoS
- Vulnerability management: Process for tracking and remediating CVEs
- Build security: Secure CI/CD pipelines, sign images, use minimal base images
Audit Logging
Audit logging records actions performed in your Kubernetes cluster, essential for security monitoring and compliance.
Audit policy levels from least to most verbose:
- None: Don't log events
- Metadata: Log request metadata (user, timestamp, resource, verb) but not request or response body
- Request: Log event metadata and request body but not response body
- RequestResponse: Log event metadata, request and response bodies
Configure the kube-apiserver with:
Compliance and Standards
Kubernetes environments may need to comply with various security standards and regulations.
Key security standards and frameworks:
- CIS Kubernetes Benchmark: Consensus-based configuration guidelines for Kubernetes security
- Covers master node, worker node, policies, and authentication configurations
- Available for different Kubernetes versions
- NIST Guidelines: National Institute of Standards and Technology security framework
- NIST SP 800-190: Application Container Security Guide
- NIST SP 800-204: Security Strategies for Microservices
- SOC 2 Compliance: Service Organization Control framework
- Focus on security, availability, processing integrity, confidentiality, and privacy
- Requires proper access controls, encryption, and monitoring
- PCI DSS Requirements: Payment Card Industry Data Security Standard
- Requirements for securing cardholder data
- Network security, encryption, access control, and monitoring requirements
- HIPAA Compliance: Health Insurance Portability and Accountability Act
- Protecting sensitive patient health information
- Requires encryption, access controls, and audit logging
- ISO 27001: International standard for information security management
- Risk-based approach to information security
- Covers people, processes, and technology
Security Monitoring
Continuous monitoring is essential for maintaining the security posture of your Kubernetes environment.
Tools and Practices
- Security scanners: Regular scanning of images and infrastructure (Trivy, Clair, Anchore)
- Log analysis: Centralized logging with anomaly detection (ELK, Splunk, Datadog)
- Intrusion detection: Runtime security monitoring (Falco, Sysdig Secure, Aqua)
- Vulnerability scanning: Continuous vulnerability assessment (Nessus, Qualys, OpenVAS)
- Compliance monitoring: Automated compliance checking (Kube-bench, OPA Gatekeeper)
- Admission controllers: Policy enforcement at admission time (PodSecurityPolicy, OPA)
- Runtime protection: Detecting and preventing suspicious behaviors
Response Plan
- Incident detection: Automated alerting based on security events and anomalies
- Response procedures: Documented steps for handling different types of security incidents
- Container compromise
- Unauthorized access
- Data exposure
- Denial of service
- Recovery process: Methods for secure recovery after an incident
- Isolation procedures
- Evidence collection
- Clean-up and restoration
- Documentation: Thorough documentation of incidents, responses, and lessons learned
- Team responsibilities: Clear roles and responsibilities during security incidents
- Incident commander
- Technical lead
- Communications lead
- Executive sponsor
Advanced Security Topics
Runtime Security
Runtime security focuses on protecting containers during execution. OpenShift's SecurityContextConstraints provide an example of advanced runtime controls:
For vanilla Kubernetes, similar controls can be implemented using Pod Security Standards and admission controllers like OPA Gatekeeper or Kyverno.
Encryption Configuration
Kubernetes can encrypt Secret resources and other sensitive data at rest in etcd:
Apply this configuration to the kube-apiserver with:
After enabling encryption, existing secrets can be encrypted with:
Security Checklist
Essential security measures:
- RBAC implementation
- Follow principle of least privilege
- Regularly audit permissions
- Use service accounts appropriately
- Network policies
- Default deny policies in each namespace
- Explicit ingress/egress rules
- Network segmentation between environments
- Pod security
- Apply Pod Security Standards
- Use non-root users
- Set resource limits
- Drop unnecessary capabilities
- Secret management
- Enable etcd encryption
- Regular secret rotation
- Consider external secret stores
- Limit secret access
- Audit logging
- Configure comprehensive audit policies
- Centralize and protect audit logs
- Regular log analysis
- Retain logs for compliance
- TLS configuration
- Encrypt all communications
- Automated certificate management
- Strong TLS ciphers
- Certificate rotation
- Container security
- Use minimal base images
- Regular vulnerability scanning
- Image signing and verification
- Multi-stage builds
- Access controls
- Strong authentication
- Multi-factor where possible
- JIT (Just-In-Time) access
- Regular access reviews
- Monitoring setup
- Runtime security monitoring
- Behavioral analytics
- Resource monitoring
- Alerting on suspicious activity
- Incident response
- Documented procedures
- Regular drills
- Post-incident analysis
- Continuous improvement