Storage

Understanding Kubernetes storage concepts, volumes, and persistence

Kubernetes provides various storage options to manage data persistence for containers and pods. Storage management in Kubernetes is designed to abstract the underlying storage infrastructure, allowing applications to consume storage resources without being tightly coupled to the specific storage technology.

The Kubernetes storage architecture is built around the concept of volumes, which provide a way for containers to access and persist data beyond their lifecycle. This architecture addresses key challenges in containerized environments, including data persistence, sharing data between containers, and managing the lifecycle of storage resources.

Storage Concepts

Volumes

Basic storage abstraction in Kubernetes
Pod-level storage that exists for the pod's lifetime
Multiple types available (emptyDir, hostPath, configMap, secret, etc.)
Lifecycle tied to pod (created when pod is created, deleted when pod is deleted)
Can be shared between containers in the same pod
Supports various backend storage systems
Used for both ephemeral and persistent storage needs
Defined in the pod specification

Persistent Volumes (PV)

Cluster-level storage resource
Independent of pods and their lifecycle
Admin provisioned or dynamically created via StorageClass
Reusable resources that can be claimed and reclaimed
Defined by capacity, access modes, storage class, reclaim policy
Represents actual physical storage in the infrastructure
Supports different volume plugins (AWS EBS, Azure Disk, NFS, etc.)
Can exist before and after pods that use them

Persistent Volume Claims (PVC)

Storage requests made by users
User abstraction that hides storage implementation details
Binds to PV that meets the requirements
Pod storage interface that pods use to request storage
Defined by storage capacity, access modes, and storage class
Acts as an intermediary between pods and PVs
Can request specific storage class or amount of storage
Enables separation of concerns between users and administrators

Volume Types

EmptyDir

An empty directory that's created when a Pod is assigned to a node and exists as long as the Pod runs on that node. When a Pod is removed, the data in the emptyDir is deleted permanently.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: nginx
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
  volumes:
  - name: cache-volume
    emptyDir: 
      medium: "" # Memory or disk-backed (use "Memory" for tmpfs)
      sizeLimit: 1Gi # Optional size limit (Kubernetes 1.18+)

Use cases for emptyDir:

Scratch space for temporary files
Checkpoint data for long computations
Sharing files between containers in a pod
Cache space for application data
Holding data processed by one container and used by another

HostPath

Mounts a file or directory from the host node's filesystem into your Pod. This type of volume presents significant security risks and should be used with caution.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test-container
    image: nginx
    volumeMounts:
    - mountPath: /test-pd
      name: test-volume
      readOnly: true # Good security practice for host mounts
  volumes:
  - name: test-volume
    hostPath:
      path: /data # Path on the host
      type: Directory # Type checks to perform before mounting
      # Other types: DirectoryOrCreate, File, FileOrCreate, Socket, CharDevice, BlockDevice

Important considerations for hostPath:

Data is not portable between nodes
Pods using the same path on different nodes will see different data
Security risk due to potential access to host filesystem
Often used for system-level pods that need access to host resources
Not suitable for most applications; prefer PVs for persistent data

Persistent Volumes

PV Definition

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-storage
  labels:
    type: local
    environment: production
spec:
  capacity:
    storage: 10Gi # The size of the volume
  volumeMode: Filesystem # Filesystem or Block
  accessModes:
    - ReadWriteOnce # RWO: Only one node can mount as read-write
    # Other options: ReadOnlyMany (ROX), ReadWriteMany (RWX)
  persistentVolumeReclaimPolicy: Retain # What happens when PVC is deleted
    # Options: Retain, Delete, Recycle (deprecated)
  storageClassName: standard # The storage class this PV belongs to
  mountOptions:
    - hard
    - nfsvers=4.1
  # hostPath example (local storage, generally for testing)
  hostPath:
    path: /data
  
  # Other backend examples:
  # AWS EBS
  # awsElasticBlockStore:
  #   volumeID: <volume-id>
  #   fsType: ext4
  
  # Azure Disk
  # azureDisk:
  #   diskName: myAKSDisk
  #   diskURI: /subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Compute/disks/<disk-name>
  #   kind: Managed
  #   fsType: ext4

PVC Definition

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-claim
  namespace: default
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
spec:
  accessModes:
    - ReadWriteOnce # Must be compatible with the PV's access modes
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi # Requested storage capacity
  storageClassName: standard # Must match the PV's storage class
  # Optional selector to bind to specific PVs
  selector:
    matchLabels:
      environment: production

Using PVC in Pod

apiVersion: v1
kind: Pod
metadata:
  name: web-server
spec:
  containers:
  - name: web
    image: nginx
    volumeMounts:
    - mountPath: "/usr/share/nginx/html"
      name: web-data
      readOnly: false
    ports:
    - containerPort: 80
      name: "http-server"
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
  volumes:
  - name: web-data
    persistentVolumeClaim:
      claimName: pvc-claim
  securityContext:
    fsGroup: 1000 # Set group ownership for mounted volumes

PV and PVC Lifecycle

Provisioning: Static (admin creates PVs) or Dynamic (via StorageClass)
Binding: PVC is bound to a suitable PV
Using: Pod uses the PVC as a volume
Reclaiming: When PVC is deleted, PV is reclaimed according to policy
- Retain: PV and data are kept (admin must clean up manually)
- Delete: PV and associated storage are deleted
- Recycle: Basic scrub (rm -rf) before reuse (deprecated)

Storage Classes

StorageClasses enable dynamic provisioning of Persistent Volumes. They abstract the underlying storage provider details and allow administrators to define different "classes" of storage with varying performance characteristics, reclaim policies, and other parameters.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: kubernetes.io/aws-ebs # The volume plugin used for provisioning
parameters:
  type: gp2 # Amazon EBS volume type
  fsType: ext4 # Filesystem type to be created
  encrypted: "true" # Enable encryption
  iopsPerGB: "10" # IOPS per GB for io1 volumes
reclaimPolicy: Retain # What happens to PVs when PVC is deleted
allowVolumeExpansion: true # Allow PVCs to be expanded after creation
volumeBindingMode: WaitForFirstConsumer # Delay binding until pod is created
# Immediate is the alternative - bind immediately when PVC is created
mountOptions:
  - debug

Different provisioner examples:

# GCE Persistent Disk
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard-gce
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  replication-type: none

# Azure Disk
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-premium
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Premium_LRS
  kind: Managed

# Local Storage
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

Access Modes

Available access modes:

ReadWriteOnce (RWO)
- Volume can be mounted as read-write by a single node
- Multiple pods on the same node can use the volume
- Most common for block storage like AWS EBS, Azure Disk, GCE PD
- Example use case: Database storage where only one instance needs access
ReadOnlyMany (ROX)
- Volume can be mounted as read-only by many nodes simultaneously
- Multiple pods across different nodes can read from the volume
- Useful for shared configuration or static content
- Supported by NFS, CephFS, some cloud providers
- Example use case: Configuration data or static website content
ReadWriteMany (RWX)
- Volume can be mounted as read-write by many nodes simultaneously
- Multiple pods across different nodes can read/write to the volume
- Less commonly supported; typically available with NFS, CephFS, GlusterFS
- Not supported by most cloud block storage (EBS, Azure Disk)
- Example use case: Shared media storage or development environments
ReadWriteOncePod (RWOP) (Kubernetes 1.22+)
- Volume can be mounted as read-write by only one pod
- Stricter than RWO which allows multiple pods on same node
- Ensures exclusive access for a single pod
- Example use case: Critical workloads requiring exclusive access

Volume Snapshots

Volume snapshots provide a way to create point-in-time copies of persistent volumes. This feature is particularly useful for backup, disaster recovery, and creating copies of data for testing or development environments.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: new-snapshot
  namespace: default
  labels:
    environment: production
    app: database
spec:
  volumeSnapshotClassName: csi-hostpath-snapclass
  source:
    persistentVolumeClaimName: pvc-claim
  # Alternative: use volumeSnapshotContentName to pre-provisioned snapshot
  # volumeSnapshotContentName: existing-snapshot-content

Volume Snapshot Class

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-hostpath-snapclass
driver: hostpath.csi.k8s.io
deletionPolicy: Delete  # Or Retain
parameters:
  # Driver-specific parameters
  csi.storage.k8s.io/snapshotter-secret-name: snapshotter-secret
  csi.storage.k8s.io/snapshotter-secret-namespace: default

Restoring from a Snapshot

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-from-snapshot
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: standard
  dataSource:
    name: new-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Snapshot operations require:

The VolumeSnapshot CRD (CustomResourceDefinition)
The snapshot controller
A CSI driver that supports snapshots
VolumeSnapshotClass configuration

Dynamic Provisioning

Storage Class

Defines storage type and characteristics
Enables automatic provisioning of storage resources
Supports different storage providers (AWS, GCP, Azure, on-premises)
Configures quality of service parameters (IOPS, throughput)
Can be set as the default for the cluster
Defines volume binding mode (immediate or wait for consumer)
Configures reclaim policy for created PVs
Enables or disables volume expansion

Automatic PV Creation

Triggered based on PVC request
Uses StorageClass to determine how to provision storage
Provider-specific parameters control the resulting storage
Handles resource management automatically
Creates appropriately sized volumes based on PVC request
Labels and annotations from StorageClass are applied to PV
Example flow:
1. User creates PVC with storageClassName
2. Dynamic provisioner watches for new PVCs
3. Provisioner creates actual storage in infrastructure
4. Provisioner creates PV object in Kubernetes
5. Kubernetes binds PVC to newly created PV
6. Pod can now use the PVC

Example Dynamic Provisioning

# First, define a StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  iopsPerGB: "3000"
  throughput: "125"
reclaimPolicy: Delete
allowVolumeExpansion: true

# Then, create a PVC that uses the StorageClass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: fast-storage

# Finally, use the PVC in a Pod
apiVersion: v1
kind: Pod
metadata:
  name: web-app
spec:
  containers:
  - name: web-app
    image: nginx
    volumeMounts:
    - mountPath: "/data"
      name: data-volume
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: dynamic-pvc

Best Practices

Use PVs and PVCs for persistence
- Separate storage concerns from application deployment
- Leverage the abstraction provided by PVCs to decouple apps from storage implementation
- Use standardized PVC requests across applications and environments
- Consider using Helm charts or operators to manage related resources together
- Example:
  # In your application template volumes: - name: data persistentVolumeClaim: claimName: {{ .Release.Name }}-data

Implement proper backup strategies

Use volume snapshots for point-in-time backups
Implement application-consistent backups where possible
Automate backup processes with CronJobs
Store backups in multiple locations/regions
Regularly test restore procedures
Consider using Velero or other Kubernetes-native backup solutions

Example CronJob for database backup:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: db-backup
spec:
  schedule: "0 2 * * *"  # Every day at 2am
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:13
            command: ["/bin/sh", "-c", "pg_dump -U postgres -h db > /backup/db-$(date +%Y%m%d).sql"]
            volumeMounts:
            - name: backup-volume
              mountPath: /backup
          volumes:
          - name: backup-volume
            persistentVolumeClaim:
              claimName: backup-pvc
          restartPolicy: OnFailure

Consider storage performance requirements
- Match storage class to application performance needs
- Use SSD-backed storage for I/O intensive workloads
- Consider IOPS, throughput, and latency requirements
- Test with realistic workloads before production
- Use different storage classes for different requirements
- Monitor I/O metrics to detect bottlenecks
- Example storage class for high-performance workloads:
  apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: high-performance provisioner: kubernetes.io/aws-ebs parameters: type: io1 iopsPerGB: "50"
Plan capacity requirements carefully
- Start with accurate capacity estimates
- Implement monitoring for storage utilization
- Configure alerts for high usage thresholds
- Use volume expansion for growing needs
- Consider storage quotas per namespace
- Document capacity planning process
- Example ResourceQuota:
  apiVersion: v1 kind: ResourceQuota metadata: name: storage-quota namespace: team-a spec: hard: persistentvolumeclaims: "10" requests.storage: "500Gi"
Use appropriate access modes
- Choose access modes based on application requirements
- Understand the limitations of your storage provider
- Use ReadWriteOnce for most stateful applications
- Use ReadWriteMany only when truly needed
- Document access mode decisions
- Example:
  # Database (single instance) - ReadWriteOnce accessModes: - ReadWriteOnce # Shared media storage - ReadWriteMany accessModes: - ReadWriteMany
Monitor storage usage and health
- Set up Prometheus monitoring for storage metrics
- Monitor PV/PVC status and events
- Track storage utilization trends
- Monitor latency and throughput
- Set up alerts for storage-related issues
- Regularly audit unused PVs and PVCs
- Example Prometheus query for PVC usage:
  kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100
Implement proper security measures
- Use encryption for sensitive data
- Implement appropriate RBAC for storage resources
- Consider using Security Contexts with fsGroup
- Secure storage provider credentials
- Use network policies to protect storage services
- Example security context:
  securityContext: fsGroup: 1000 runAsUser: 1000 runAsGroup: 1000
Consider disaster recovery scenarios
- Design for zone and region failures
- Implement cross-region backup strategies
- Document and test recovery procedures
- Consider using storage replication where available
- Define Recovery Time Objective (RTO) and Recovery Point Objective (RPO)

Common Storage Providers

Cloud Providers

AWS EBS (Elastic Block Store)
- Block storage for AWS EC2 instances
- Provides ReadWriteOnce access mode
- Multiple volume types (gp3, io2, st1, sc1)
- Supports snapshots and encryption
- Region-specific and availability zone bound
- Example:
  apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ebs-sc provisioner: kubernetes.io/aws-ebs parameters: type: gp3 encrypted: "true"
Azure Disk
- Block storage for Azure
- Provides ReadWriteOnce access
- Multiple SKUs (Standard_LRS, Premium_LRS, UltraSSD_LRS)
- Supports managed and unmanaged disks
- Example:
  apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: azure-disk provisioner: kubernetes.io/azure-disk parameters: storageaccounttype: Premium_LRS kind: Managed
Google Persistent Disk
- Block storage for GCP
- pd-standard (HDD) and pd-ssd options
- Regional or zonal deployment
- Automatic encryption
- Example:
  apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gce-pd provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd replication-type: none
OpenStack Cinder
- Block storage for OpenStack
- Various volume types based on configuration
- Integration with OpenStack authentication
- Example:
  apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: cinder-sc provisioner: kubernetes.io/cinder parameters: type: fast availability: nova

On-Premise

NFS (Network File System)
- Provides ReadWriteMany access
- Good for shared file access
- Widely supported in enterprise environments
- Simple to set up and manage
- Example:
  apiVersion: v1 kind: PersistentVolume metadata: name: nfs-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany nfs: server: nfs-server.example.com path: "/shared"
iSCSI (Internet Small Computer Systems Interface)
- Block storage protocol
- Widely supported in enterprise storage
- Provides ReadWriteOnce access
- Requires iSCSI initiator configuration
- Example:
  apiVersion: v1 kind: PersistentVolume metadata: name: iscsi-pv spec: capacity: storage: 20Gi accessModes: - ReadWriteOnce iscsi: targetPortal: 10.0.0.1:3260 iqn: iqn.2000-01.com.example:storage.kube.sys1.xyz lun: 0 fsType: ext4 readOnly: false
Ceph
- Distributed storage system
- Provides block (RBD), file (CephFS), and object storage
- Highly scalable and resilient
- Supports ReadWriteMany with CephFS
- Example RBD (Rados Block Device):
  apiVersion: v1 kind: PersistentVolume metadata: name: ceph-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce rbd: monitors: - 10.16.154.78:6789 - 10.16.154.82:6789 pool: kube image: mysql-persistent user: admin secretRef: name: ceph-secret fsType: ext4 readOnly: false
GlusterFS
- Distributed file system
- Provides ReadWriteMany access
- Scales horizontally
- Good for large file storage
- Example:
  apiVersion: v1 kind: PersistentVolume metadata: name: glusterfs-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany glusterfs: endpoints: "glusterfs-cluster" path: "volume1" readOnly: false

CSI Drivers (Container Storage Interface)

Standardized interface for storage providers
Enables third-party storage systems to work with Kubernetes
Dynamically provisioned volumes
Support for advanced features like snapshots
Plugin model for extending storage capabilities
Examples include:
- Dell EMC PowerFlex
- NetApp Trident
- Pure Storage Pure Service Orchestrator
- Portworx
- VMware vSphere CSI Driver

Troubleshooting

Common issues and solutions:

PVC stuck in pending state
- Issue: PVC remains in pending state and doesn't bind to a PV
- Diagnosis:
  kubectl get pvc kubectl describe pvc <pvc-name> kubectl get events --field-selector involvedObject.name=<pvc-name>
- Common causes:
  - No matching PV available
  - StorageClass doesn't exist
  - Access mode incompatibility
  - Capacity requirements not met
  - Volume binding mode set to WaitForFirstConsumer
- Solution:
  - Check if the StorageClass exists and has a provisioner
  - Verify PVC is requesting supported access modes
  - Check if dynamic provisioning is enabled
  - Ensure cloud provider permissions are correct
  - Create a matching PV manually if using static provisioning
Volume mount failures
- Issue: Pod can't start because of volume mount problems
- Diagnosis:
  kubectl describe pod <pod-name> kubectl get events
- Common causes:
  - PV is mounted on another node (for RWO volumes)
  - Filesystem issues or corruption
  - Wrong permissions on the mounted volume
  - Network issues with NFS or other network storage
- Solution:
  - Check if volume is already mounted elsewhere
  - Verify storage backend health and connectivity
  - Check filesystem with fsck if applicable
  - Ensure PV and node are in the same zone for zonal storage
Permission issues
- Issue: Container can't write to mounted volume
- Diagnosis:
  kubectl exec -it <pod-name> -- ls -la /mount/path kubectl exec -it <pod-name> -- id
- Common causes:
  - Mismatched user/group IDs
  - Read-only filesystem
  - SELinux/AppArmor restrictions
- Solution:
  - Add securityContext with appropriate fsGroup
  - Use an initContainer to set permissions
  - Modify container to run as the correct user
  - Example:
    securityContext: fsGroup: 2000 runAsUser: 1000
Storage capacity issues
- Issue: Insufficient storage space on PV
- Diagnosis:
  kubectl exec -it <pod-name> -- df -h kubectl get pv <pv-name> -o yaml | grep capacity
- Common causes:
  - Initial capacity too small
  - Application consuming more space than expected
  - Temporary files not being cleaned up
  - Log files growing unchecked
- Solution:
  - Use volume expansion if supported
  - Implement log rotation and cleanup routines
  - Move data to a larger volume
  - Configure monitoring and alerts for storage usage

Storage Management

Volume Expansion

Volume expansion allows you to increase the size of a PVC without disrupting applications:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi # Increase this value
  storageClassName: standard

Steps for volume expansion:

Verify StorageClass supports expansion (allowVolumeExpansion: true)
Edit the PVC to increase storage request
Wait for resize to complete (may require pod restart depending on storage provider)
Verify new size is available to the pod

# Check if expansion is supported
kubectl get sc <storage-class> -o yaml | grep allowVolumeExpansion

# Edit PVC to resize
kubectl edit pvc myclaim
# Then increase storage value

# Check expansion status
kubectl describe pvc myclaim

Data Migration

When you need to move data to a different storage class or region:

Create volume snapshot of the source PVC
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: source-snapshot spec: volumeSnapshotClassName: csi-snapclass source: persistentVolumeClaimName: source-pvc

Create new PV from snapshot

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: new-pvc
spec:
  storageClassName: new-storage-class
  dataSource:
    name: source-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Update pod configuration to use the new PVC
volumes: - name: data-volume persistentVolumeClaim: claimName: new-pvc # Changed from source-pvc
Verify data integrity after migration

StatefulSet Volume Management

StatefulSets have special handling for persistent storage:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.20
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "standard"
      resources:
        requests:
          storage: 1Gi

The volumeClaimTemplates creates a PVC for each pod in the StatefulSet with predictable names: www-web-0, www-web-1, etc.

Monitoring and Maintenance

Regular Tasks

Monitor usage: Set up Prometheus metrics for volume utilization
# Example Prometheus query for PVC usage kubelet_volume_stats_used_bytes{namespace="default"} / kubelet_volume_stats_capacity_bytes{namespace="default"} * 100
Check capacity: Implement automated checks for storage thresholds
# Script to alert on volumes over 80% full kubectl get pvc --all-namespaces -o json | jq '.items[] | select(.status.phase=="Bound") | .metadata.name + " " + .metadata.namespace + " " + .status.capacity.storage'
Verify backups: Regularly test restore procedures from snapshots
# Create test restore from snapshot kubectl apply -f test-restore-pvc.yaml kubectl apply -f test-restore-pod.yaml kubectl exec -it test-restore-pod -- ls -la /data
Update policies: Review and adjust retention policies and quotas
# Update storage quota kubectl apply -f updated-storage-quota.yaml
Performance analysis: Monitor I/O metrics for storage bottlenecks
# Install storage performance monitoring tools helm install storage-monitor prometheus-community/prometheus

Health Checks

Volume status: Regularly check PV and PVC status
kubectl get pv,pvc --all-namespaces
Mount points: Verify volumes are correctly mounted in pods
kubectl exec -it <pod-name> -- df -h
I/O performance: Monitor read/write latency and throughput
kubectl exec -it <pod-name> -- fio --name=benchmark --ioengine=libaio --rw=randrw --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --group_reporting
Error logs: Check for volume-related errors in pod and system logs
kubectl logs <pod-name> | grep -i "volume\|storage\|disk\|i/o"
Storage events: Monitor Kubernetes events for storage issues
kubectl get events --field-selector involvedObject.kind=PersistentVolume,involvedObject.kind=PersistentVolumeClaim

Automated Storage Management

Create automated processes for common storage tasks:

# Example CronJob for regular volume snapshots
apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-volume-snapshot
spec:
  schedule: "0 1 * * *"  # 1:00 AM every day
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: snapshot-creator
          containers:
          - name: snapshot-creator
            image: bitnami/kubectl:latest
            command:
            - /bin/sh
            - -c
            - |
              kubectl apply -f /snapshots/create-snapshot.yaml
              echo "Created snapshot at $(date)"
          restartPolicy: OnFailure

Local Persistent Volumes

For high-performance workloads or specific hardware requirements, Kubernetes supports local persistent volumes that are directly attached to a specific node:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-pv
spec:
  capacity:
    storage: 100Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /mnt/disks/ssd1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-node-1

Key considerations for local volumes:

Data is not replicated or protected at the storage level
Pods using local volumes will be scheduled to specific nodes
If the node fails, pod can't be rescheduled elsewhere
Use for high-performance requirements or data that can be regenerated
Consider using StatefulSets with anti-affinity for distributed workloads

Advanced Storage Patterns

Sidecar Containers for Storage Management

apiVersion: v1
kind: Pod
metadata:
  name: log-collection
spec:
  containers:
  - name: app
    image: my-app
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
  - name: log-collector
    image: log-collector
    volumeMounts:
    - name: logs
      mountPath: /var/log/app
      readOnly: true
  volumes:
  - name: logs
    emptyDir: {}

Init Containers for Data Preparation

apiVersion: v1
kind: Pod
metadata:
  name: data-init
spec:
  initContainers:
  - name: data-downloader
    image: busybox
    command: ["wget", "-O", "/data/input.dat", "https://example.com/data"]
    volumeMounts:
    - name: data
      mountPath: /data
  containers:
  - name: app
    image: data-processor
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-pvc

Ephemeral Volumes

For non-persistent data that should exist for the pod's lifetime but requires more flexibility than emptyDir:

apiVersion: v1
kind: Pod
metadata:
  name: ephemeral-demo
spec:
  containers:
  - name: app
    image: my-app
    volumeMounts:
    - name: scratch
      mountPath: /scratch
  volumes:
  - name: scratch
    ephemeral:
      volumeClaimTemplate:
        metadata:
          labels:
            type: ephemeral
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "scratch-storage"
          resources:
            requests:
              storage: 1Gi

Edit this page

Networking

Understanding Kubernetes networking concepts and implementation

Security

Understanding Kubernetes security concepts, best practices, and implementations