Welcome to from-docker-to-kubernetes

Operators & CRDs

Understanding Kubernetes Operators, Custom Resources, and extending Kubernetes

H A R S H H A A
@NotHarshhaa

Kubernetes Operators

Operators are software extensions to Kubernetes that use custom resources to manage applications and their components. They implement domain-specific knowledge to automate the entire lifecycle of the software they manage, from deployment and configuration to updates, backups, and failure handling.

Operators follow the Kubernetes principle of reconciliation loops - continuously comparing the desired state with the actual state and taking actions to align them. This makes them powerful tools for automating complex operational tasks that would otherwise require manual intervention.

Understanding Custom Resources

Custom Resource Definitions (CRDs)

  • Extend Kubernetes API: Add new endpoints to the Kubernetes API server
  • Define new resource types: Create schema and validation for custom objects
  • Domain-specific objects: Represent application-specific concepts as Kubernetes resources
  • Declarative management: Apply, update, and delete with standard kubectl commands
  • Kubernetes-native interfaces: Integrate with existing tools and workflows
  • Versioning support: Enable API evolution with multiple versions
  • Namespace or cluster scoped: Control resource visibility and isolation

Custom Controllers

  • Watch custom resources: Monitor the Kubernetes API for changes to custom objects
  • Implement business logic: Encode domain knowledge and operational procedures
  • Reconcile desired state: Continuously work to make actual state match specification
  • Manage application lifecycle: Handle creation, updates, scaling, and deletion
  • Automate operational tasks: Perform backups, upgrades, failovers, and more
  • Handle edge cases: Implement retry logic and error handling
  • Report status: Update status subresource with current conditions

The combination of CRDs and controllers is what makes Operators powerful. CRDs define the "what" (the desired state) while controllers implement the "how" (the reconciliation logic).

Creating Custom Resources

A CustomResourceDefinition (CRD) is a Kubernetes resource that defines a new type of custom resource. Here's a detailed example of a CRD for a simple CronTab resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com  # Must be <plural>.<group>
spec:
  # The API group this resource belongs to
  group: stable.example.com
  
  # Different versions of the API
  versions:
    - name: v1                   # Version name
      served: true               # Whether this version is available through the API
      storage: true              # Whether this version is used for storage in etcd
      
      # OpenAPI v3 schema for validation
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                  description: "Cron expression for the job schedule"
                  pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'  # Regex validation
                image:
                  type: string
                  description: "Container image to run"
                replicas:
                  type: integer
                  description: "Number of replicas to run"
                  minimum: 1     # Validation constraint
              required: ["cronSpec", "image"]  # Required fields
              
      # Subresources for this version
      subresources:
        # Enable status subresource
        status: {}
        
      # Additional printer columns for kubectl get
      additionalPrinterColumns:
      - name: Schedule
        type: string
        description: The cron schedule
        jsonPath: .spec.cronSpec
      - name: Age
        type: date
        jsonPath: .metadata.creationTimestamp
        
  # Whether this resource is namespaced or cluster-scoped
  scope: Namespaced
  
  # Naming configuration
  names:
    plural: crontabs        # Plural name used in URLs
    singular: crontab       # Singular name
    kind: CronTab           # Kind name for YAML/JSON
    shortNames:             # Short aliases for kubectl
    - ct

This CRD includes:

  • Proper versioning support for API evolution
  • OpenAPI schema validation to ensure correctness
  • Subresource support for clean separation of status updates
  • Custom printer columns for better kubectl output
  • Pattern validation for the cron expression

Using Custom Resources

Once a CRD is defined, you can create instances of your custom resource. These are used just like built-in Kubernetes resources with kubectl:

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-crontab
  namespace: default
  labels:
    app: cron-service
    environment: production
  annotations:
    description: "Runs a service every 5 minutes"
spec:
  cronSpec: "*/5 * * * *"  # Run every 5 minutes
  image: my-cron-image:v1.2.3
  replicas: 3

You can manage these resources with familiar kubectl commands:

# Create the custom resource
kubectl apply -f my-crontab.yaml

# Get all CronTab resources
kubectl get crontabs

# Describe a specific CronTab
kubectl describe crontab my-crontab

# Delete a CronTab
kubectl delete crontab my-crontab

The controller for this CRD would watch for these resources and create the necessary Kubernetes objects (like Jobs or CronJobs) based on the specification.

Operator Pattern

The Operator pattern is particularly valuable for stateful applications that require specific domain knowledge to operate correctly. Traditional deployment tools may struggle with databases, message queues, and other stateful systems, but Operators can encode the necessary operational procedures directly.

An Operator can handle tasks such as:

  • Automated backup and restore procedures
  • Data replication configuration
  • Leader election in clustered applications
  • Rolling updates with zero downtime
  • Scaling with proper data rebalancing
  • Disaster recovery processes

Popular Operators

Prometheus Operator

  • Manages Prometheus instances: Deploys and configures Prometheus servers
  • Configures monitoring targets: Automatically discovers and scrapes services
  • Handles alerting rules: Manages AlertManager and notification policies
  • Manages Grafana dashboards: Provisions dashboards and data sources
  • Simplifies monitoring setup: Creates ServiceMonitors for target discovery
  • Handles high availability: Supports Prometheus clustering for reliability
  • Implements sharding: Scales monitoring across multiple instances

PostgreSQL Operator

  • Deploys PostgreSQL clusters: Creates primary and replica instances
  • Handles high availability: Manages automatic failover mechanisms
  • Manages backups: Schedules and restores point-in-time backups
  • Implements scaling: Adjusts resources and replica count based on demand
  • Automates upgrades: Performs zero-downtime version upgrades
  • Manages connection pooling: Configures PgBouncer for optimal performance
  • Monitors database health: Collects metrics and alerts on issues
  • Manages users and permissions: Handles role-based access control

Elasticsearch Operator

  • Creates Elasticsearch clusters: Provisions multi-node Elasticsearch deployments
  • Manages Kibana instances: Deploys and configures visualization frontend
  • Handles data nodes: Distributes data for performance and redundancy
  • Implements security: Configures authentication, authorization, and TLS
  • Automates operations: Handles index management and shard allocation
  • Manages topology: Places nodes across availability zones
  • Handles snapshots: Configures automated backup schedules
  • Upgrades safely: Performs rolling updates of cluster components

Other notable operators include:

  • Strimzi Kafka Operator: Manages Apache Kafka clusters, topics, users, and more
  • MongoDB Operator: Automates MongoDB replica sets and sharded clusters
  • Redis Operator: Manages Redis Sentinel and Redis Cluster deployments
  • Jaeger Operator: Manages distributed tracing infrastructure
  • Vault Operator: Automates HashiCorp Vault deployment and secret management
  • Istio Operator: Simplifies service mesh installation and upgrades

Operator Frameworks

Several frameworks exist to simplify operator development, with Operator SDK being the most widely used. These frameworks provide scaffolding, utilities, and best practices to accelerate development.

Operator SDK

The Operator SDK, part of the Operator Framework, supports multiple options for implementing operators:

  1. Go: Native language for Kubernetes with direct client-go integration
  2. Ansible: For teams with existing Ansible expertise
  3. Helm: For converting existing Helm charts into operators
# Install Operator SDK CLI
export ARCH=$(case $(uname -m) in x86_64) echo -n amd64 ;; aarch64) echo -n arm64 ;; *) echo -n $(uname -m) ;; esac)
export OS=$(uname | awk '{print tolower($0)}')
export OPERATOR_SDK_DL_URL=https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0
curl -LO ${OPERATOR_SDK_DL_URL}/operator-sdk_${OS}_${ARCH}
chmod +x operator-sdk_${OS}_${ARCH} && sudo mv operator-sdk_${OS}_${ARCH} /usr/local/bin/operator-sdk

# Create new operator project with Go
operator-sdk init --domain example.com --repo github.com/example/memcached-operator

# Create API and controller
operator-sdk create api --group cache --version v1 --kind Memcached --resource --controller

# Generate CRD manifests
make manifests

# Build and push the operator image
make docker-build docker-push IMG=quay.io/example/memcached-operator:v0.0.1

# Deploy the operator to a cluster
make deploy IMG=quay.io/example/memcached-operator:v0.0.1

Other Operator Development Options

KOPF (Kubernetes Operator Pythonic Framework):

  • Python-based framework for writing operators
  • Event-driven programming model
  • Simpler learning curve for Python developers

Kubebuilder:

  • Foundation for Operator SDK's Go support
  • Focused specifically on Go-based operators
  • Uses controller-runtime library

KUDO (Kubernetes Universal Declarative Operator):

  • Declarative approach to building operators
  • YAML-based operator definitions
  • No programming required for many use cases

Operator Lifecycle Manager (OLM)

Operator Lifecycle Manager (OLM) helps cluster administrators manage the lifecycle of operators in a Kubernetes cluster, from installation to updates to removal.

Benefits

  • Operator discoverability: Catalog of available operators in a cluster
  • Dependency resolution: Automatically installs operator dependencies
  • Cluster stability: Ensures compatibility between operators
  • Update management: Handles operator upgrades safely
  • Version tracking: Manages multiple versions of operators
  • Channel-based updates: Supports concepts like stable/beta/alpha channels
  • Namespace tenancy: Controls operator visibility and access
  • Seamless upgrades: Updates operators without service interruption

Installation

# Install OLM
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.20.0/install.sh | bash -s v0.20.0

# Verify OLM installation
kubectl get pods -n olm

# List available operators in the catalog
kubectl get packagemanifests -n olm

# Install an operator using OLM
cat <<EOF | kubectl apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: prometheus
  namespace: operators
spec:
  channel: beta
  name: prometheus
  source: operatorhubio-catalog
  sourceNamespace: olm
EOF

OLM introduces several custom resources for operator management:

  1. ClusterServiceVersion (CSV): Represents a specific version of an operator
  2. InstallPlan: Calculated list of resources to be created for an operator
  3. Subscription: Keeps operators updated by tracking a channel in a package
  4. CatalogSource: Repository of operator metadata that OLM can query
  5. OperatorGroup: Defines the service account permissions for operators

These resources work together to provide a comprehensive operator management solution for cluster administrators.

Building an Operator

Building an effective operator requires deep understanding of both Kubernetes internals and the application domain. The best operators combine Kubernetes expertise with application-specific operational knowledge.

Example: Simple Operator

Below is a more detailed example of a Go-based controller implementation for a Memcached operator:

// Example controller implementation in Go
package controllers

import (
    "context"
    "fmt"
    "reflect"

    appsv1 "k8s.io/api/apps/v1"
    corev1 "k8s.io/api/core/v1"
    "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/api/resource"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
    "sigs.k8s.io/controller-runtime/pkg/log"

    cachev1 "github.com/example/memcached-operator/api/v1"
)

// MemcachedReconciler reconciles a Memcached object
type MemcachedReconciler struct {
    client.Client
    Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=cache.example.com,resources=memcacheds/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch

// Reconcile handles Memcached CR events
func (r *MemcachedReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)
    log.Info("Starting reconciliation", "request", req)

    // Fetch the Memcached instance
    memcached := &cachev1.Memcached{}
    err := r.Get(ctx, req.NamespacedName, memcached)
    if err != nil {
        if errors.IsNotFound(err) {
            // Object not found, likely deleted - return without error
            log.Info("Memcached resource not found, ignoring since it's probably deleted")
            return ctrl.Result{}, nil
        }
        // Error reading the object - requeue the request
        log.Error(err, "Failed to get Memcached")
        return ctrl.Result{}, err
    }

    // Check if the Memcached instance is marked for deletion
    if !memcached.ObjectMeta.DeletionTimestamp.IsZero() {
        // The object is being deleted, handle any finalizers here
        return ctrl.Result{}, nil
    }

    // Define the desired Deployment object
    deployment := r.deploymentForMemcached(memcached)
    
    // Set Memcached instance as the owner and controller
    if err := controllerutil.SetControllerReference(memcached, deployment, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for deployment")
        return ctrl.Result{}, err
    }

    // Check if the Deployment already exists
    found := &appsv1.Deployment{}
    err = r.Get(ctx, types.NamespacedName{Name: deployment.Name, Namespace: deployment.Namespace}, found)
    
    if err != nil && errors.IsNotFound(err) {
        // Create the Deployment if it doesn't exist
        log.Info("Creating a new Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
        err = r.Create(ctx, deployment)
        if err != nil {
            log.Error(err, "Failed to create new Deployment")
            return ctrl.Result{}, err
        }
        // Deployment created successfully - return and requeue
        return ctrl.Result{Requeue: true}, nil
    } else if err != nil {
        log.Error(err, "Failed to get Deployment")
        return ctrl.Result{}, err
    }

    // Ensure the deployment size is as specified
    size := memcached.Spec.Size
    if *found.Spec.Replicas != size {
        found.Spec.Replicas = &size
        log.Info("Updating Deployment size", "old", *found.Spec.Replicas, "new", size)
        err = r.Update(ctx, found)
        if err != nil {
            log.Error(err, "Failed to update Deployment", "Deployment.Namespace", found.Namespace, "Deployment.Name", found.Name)
            return ctrl.Result{}, err
        }
        // Updated successfully - return and requeue
        return ctrl.Result{Requeue: true}, nil
    }

    // Create or update the Service
    service := r.serviceForMemcached(memcached)
    if err := controllerutil.SetControllerReference(memcached, service, r.Scheme); err != nil {
        log.Error(err, "Failed to set controller reference for service")
        return ctrl.Result{}, err
    }
    
    // Check if the Service already exists
    foundSvc := &corev1.Service{}
    err = r.Get(ctx, types.NamespacedName{Name: service.Name, Namespace: service.Namespace}, foundSvc)
    if err != nil && errors.IsNotFound(err) {
        // Create the Service
        log.Info("Creating a new Service", "Service.Namespace", service.Namespace, "Service.Name", service.Name)
        err = r.Create(ctx, service)
        if err != nil {
            log.Error(err, "Failed to create new Service")
            return ctrl.Result{}, err
        }
    } else if err != nil {
        log.Error(err, "Failed to get Service")
        return ctrl.Result{}, err
    }

    // List the pods for this memcached's deployment
    podList := &corev1.PodList{}
    listOpts := []client.ListOption{
        client.InNamespace(memcached.Namespace),
        client.MatchingLabels(map[string]string{"app": "memcached", "memcached_cr": memcached.Name}),
    }
    if err = r.List(ctx, podList, listOpts...); err != nil {
        log.Error(err, "Failed to list pods")
        return ctrl.Result{}, err
    }
    
    // Update the Memcached status with the pod names
    podNames := getPodNames(podList.Items)
    if !reflect.DeepEqual(podNames, memcached.Status.Nodes) {
        memcached.Status.Nodes = podNames
        err := r.Status().Update(ctx, memcached)
        if err != nil {
            log.Error(err, "Failed to update Memcached status")
            return ctrl.Result{}, err
        }
    }

    log.Info("Reconciliation complete")
    return ctrl.Result{}, nil
}

// deploymentForMemcached returns a memcached Deployment object
func (r *MemcachedReconciler) deploymentForMemcached(m *cachev1.Memcached) *appsv1.Deployment {
    ls := map[string]string{
        "app":          "memcached",
        "memcached_cr": m.Name,
    }
    
    replicas := m.Spec.Size
    
    dep := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: ls,
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: ls,
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Image: m.Spec.Image,
                        Name:  "memcached",
                        Ports: []corev1.ContainerPort{{
                            ContainerPort: 11211,
                            Name:          "memcached",
                        }},
                        Resources: corev1.ResourceRequirements{
                            Limits: corev1.ResourceList{
                                corev1.ResourceCPU:    resource.MustParse("200m"),
                                corev1.ResourceMemory: resource.MustParse("512Mi"),
                            },
                            Requests: corev1.ResourceList{
                                corev1.ResourceCPU:    resource.MustParse("100m"),
                                corev1.ResourceMemory: resource.MustParse("256Mi"),
                            },
                        },
                        LivenessProbe: &corev1.Probe{
                            ProbeHandler: corev1.ProbeHandler{
                                TCPSocket: &corev1.TCPSocketAction{
                                    Port: intstr.FromInt(11211),
                                },
                            },
                            InitialDelaySeconds: 30,
                            TimeoutSeconds:      5,
                        },
                        ReadinessProbe: &corev1.Probe{
                            ProbeHandler: corev1.ProbeHandler{
                                TCPSocket: &corev1.TCPSocketAction{
                                    Port: intstr.FromInt(11211),
                                },
                            },
                            InitialDelaySeconds: 5,
                            TimeoutSeconds:      3,
                        },
                    }},
                },
            },
        },
    }
    
    return dep
}

// serviceForMemcached returns a memcached Service object
func (r *MemcachedReconciler) serviceForMemcached(m *cachev1.Memcached) *corev1.Service {
    ls := map[string]string{
        "app":          "memcached",
        "memcached_cr": m.Name,
    }
    
    svc := &corev1.Service{
        ObjectMeta: metav1.ObjectMeta{
            Name:      m.Name,
            Namespace: m.Namespace,
        },
        Spec: corev1.ServiceSpec{
            Selector: ls,
            Ports: []corev1.ServicePort{{
                Port:       11211,
                TargetPort: intstr.FromInt(11211),
                Name:       "memcached",
            }},
        },
    }
    
    return svc
}

// getPodNames returns the pod names of the array of pods passed in
func getPodNames(pods []corev1.Pod) []string {
    var podNames []string
    for _, pod := range pods {
        podNames = append(podNames, pod.Name)
    }
    return podNames
}

// Setup

## Operator Capabilities

Operators can be categorized by their capability levels, from basic installation to full auto-pilot features. These levels help teams understand the maturity and functionality of an operator.

::steps
### Level 1: Basic Install
- **Installation**: Automated deployment of the application
- **Configuration**: Basic configuration options via CRD properties
- **Updates**: Simple version upgrades with minimal disruption
- **Examples**: Simple stateless applications, basic web servers
- **Complexity**: Low, primarily focused on deployment automation
- **Operational burden reduced**: Initial setup and basic updates

### Level 2: Seamless Upgrades
- **Version management**: Handling multiple versions with upgrade paths
- **Backup**: Basic backup procedures before upgrades
- **Restore**: Ability to restore from backups if upgrades fail
- **Examples**: Basic databases with backup capabilities, messaging systems
- **Complexity**: Medium-low, focuses on maintaining application state during changes
- **Operational burden reduced**: Version management, routine maintenance

### Level 3: Full Lifecycle
- **Scaling**: Dynamic scaling based on metrics or manual requests
- **Failover**: Automatic handling of node or pod failures
- **Self-healing**: Detection and remediation of common failure modes
- **Advanced configuration**: Complex application-specific settings
- **Examples**: Production databases, distributed systems, stateful applications
- **Complexity**: Medium, requires domain-specific operational knowledge
- **Operational burden reduced**: Day-to-day operations, incident response

### Level 4: Deep Insights
- **Metrics**: Comprehensive application-specific metrics collection
- **Alerts**: Intelligent alerting based on application behavior
- **Log processing**: Aggregation and analysis of application logs
- **Advanced monitoring**: Dashboards and visualization of application state
- **Examples**: Complex microservice architectures, data processing systems
- **Complexity**: Medium-high, requires understanding of application internals
- **Operational burden reduced**: Monitoring, troubleshooting, diagnostics

### Level 5: Auto Pilot
- **Horizontal/vertical scaling**: Automatic scaling based on workload patterns
- **Tuning**: Self-optimizing configuration based on usage patterns
- **Anomaly detection**: Identifying unusual behavior and self-correcting
- **Predictive maintenance**: Addressing issues before they impact users
- **Capacity planning**: Forecasting resource needs and adapting proactively
- **Examples**: Advanced data platforms, mission-critical enterprise systems
- **Complexity**: High, requires sophisticated algorithms and deep application knowledge
- **Operational burden reduced**: Performance optimization, capacity planning
::

As operators progress through these capability levels, they encapsulate more operational knowledge and reduce the manual effort required to manage complex applications. The highest levels represent a state where human operators rarely need to intervene in routine operations.

## Best Practices

::alert{type="info"}
1. **Focus on one application domain**
   - Each operator should manage a single application or tightly coupled set of components
   - Avoid creating "mega-operators" that try to manage too many different systems
   - Split complex applications into logical operator boundaries

2. **Use declarative APIs**
   - Design CRDs to specify the desired state, not the actions to take
   - Avoid imperative fields that trigger one-time actions
   - Make all operations idempotent and repeatable

3. **Minimize reconciliation time**
   - Keep reconciliation loops fast and efficient
   - Use status conditions to track long-running operations
   - Implement backoff for retries and avoid overloading the API server
   - Consider using finalizers for proper cleanup of resources

4. **Implement proper status updates**
   - Use status subresource for all status updates
   - Include detailed conditions with reason, message, and timestamps
   - Reflect the actual state of managed resources
   - Make status useful for both humans and automation

5. **Handle errors gracefully**
   - Implement comprehensive error handling
   - Use Kubernetes events to record errors and actions
   - Add detailed logging with appropriate severity levels
   - Implement retry logic with exponential backoff for transient failures

6. **Consider upgrades from the start**
   - Design CRDs with versioning in mind
   - Implement conversion webhooks for seamless upgrades
   - Test upgrades from previous versions thoroughly
   - Document upgrade paths and procedures

7. **Document operator behavior**
   - Clearly document all CRD fields and their effects
   - Explain reconciliation logic and expected behavior
   - Provide troubleshooting guides for common issues
   - Include examples for different use cases

8. **Implement proper validations**
   - Use OpenAPI validation in CRD schema
   - Implement admission webhooks for complex validations
   - Validate early to prevent invalid states
   - Provide clear error messages for validation failures
   
9. **Follow security best practices**
   - Use least privilege RBAC permissions
   - Secure sensitive data with Kubernetes Secrets
   - Implement proper TLS for all components
   - Regular security audits and updates
   
10. **Design for observability**
    - Expose Prometheus metrics for operator performance
    - Implement structured logging
    - Create default dashboards and alerts
    - Make debugging information accessible
::

## Advanced CRD Features

Kubernetes CRDs offer many advanced features that can enhance the user experience and functionality of your custom resources:

```yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames:
    - db
    categories:      # Grouping for kubectl get
    - all
    - data-stores
  scope: Namespaced  # Could also be Cluster
  versions:
  - name: v1
    served: true     # This version can be used by clients
    storage: true    # This version is persisted in etcd
    # Status subresource allows status updates without modifying spec
    subresources:
      status: {}
      # Scale subresource enables HPA integration
      scale:
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas
        labelSelectorPath: .status.labelSelector
    # Columns shown in kubectl get output
    additionalPrinterColumns:
    - name: Replicas
      type: integer
      jsonPath: .spec.replicas
      description: "Number of database replicas"
    - name: Status
      type: string
      jsonPath: .status.phase
      description: "Current status of the database"
    - name: Age
      type: date
      jsonPath: .metadata.creationTimestamp
    # Schema validation using OpenAPI v3
    schema:
      openAPIV3Schema:
        type: object
        required: ["spec"]
        properties:
          spec:
            type: object
            required: ["engine", "version", "storage"]
            properties:
              engine:
                type: string
                enum: ["postgresql", "mysql", "mongodb"]
                description: "Database engine type"
              version:
                type: string
                pattern: '^[0-9]+\.[0-9]+(\.[0-9]+)?$'
                description: "Engine version in semver format"
              replicas:
                type: integer
                minimum: 1
                maximum: 10
                default: 1
                description: "Number of database replicas"
              storage:
                type: object
                required: ["size"]
                properties:
                  size:
                    type: string
                    pattern: '^[0-9]+(Gi|Mi)$'
                    description: "Storage size (e.g. 10Gi)"
                  storageClass:
                    type: string
                    description: "Kubernetes storage class name"
              credentials:
                type: object
                properties:
                  secretName:
                    type: string
                    description: "Secret containing database credentials"
          status:
            type: object
            properties:
              phase:
                type: string
                enum: ["Pending", "Creating", "Running", "Failed"]
              message:
                type: string
              labelSelector:
                type: string
              conditions:
                type: array
                items:
                  type: object
                  required: ["type", "status"]
                  properties:
                    type:
                      type: string
                    status:
                      type: string
                      enum: ["True", "False", "Unknown"]
                    lastTransitionTime:
                      type: string
                      format: date-time
                    reason:
                      type: string
                    message:
                      type: string
    # Conversion webhook for version conversion (if you have multiple versions)
    conversion:
      strategy: Webhook
      webhook:
        conversionReviewVersions: ["v1"]
        clientConfig:
          service:
            namespace: system
            name: webhook-service
            path: /convert

Advanced CRD features enhance your operator in multiple ways:

  1. Subresources separate concerns and enable standard Kubernetes features:
    • Status subresource: Updates status without modifying spec
    • Scale subresource: Enables HPA integration
  2. Printer columns improve the CLI experience:
    • Customized kubectl get output
    • Relevant information at a glance
    • Better operational visibility
  3. Validation schema ensures data integrity:
    • Type checking and format validation
    • Required fields enforcement
    • Enumerated value restrictions
    • Numeric range constraints
    • Regex pattern validation
  4. Conversion webhooks enable API evolution:
    • Seamless version upgrades
    • Data transformation between versions
    • Backward compatibility support

Troubleshooting Operators

When operators aren't behaving as expected, systematic troubleshooting approaches can help identify and resolve issues.

Common Issues

  • Controller not reconciling:
    • Operator pod might be failing or restarting
    • Watch might not be set up correctly
    • Reconcile function might have errors
    • Resource might be ignored due to owner reference filtering
    • Webhook might be rejecting changes
  • Status not updating:
    • Missing RBAC permissions for status subresource
    • Errors in status update code
    • Status updates being overwritten by another controller
    • Custom resource not defining status fields properly
    • Status updates being throttled by API server
  • CRD validation errors:
    • Schema validation rejecting valid resources
    • Required fields missing in custom resources
    • Data type mismatches
    • Pattern validation too strict
    • Enum values not covering all cases
  • Permissions problems:
    • Insufficient RBAC for accessing resources
    • Missing service account configuration
    • Namespace restrictions
    • Security contexts preventing operations
    • Missing API groups in ClusterRole
  • Resource conflicts:
    • Multiple controllers managing same resources
    • ResourceVersion conflicts during updates
    • Finalizers preventing deletion
    • Ownership conflicts
    • Race conditions in concurrent reconciliations

Debugging

# Check operator logs with increased verbosity
kubectl logs -n operators deploy/my-operator -c manager --tail=100

# Stream logs in real-time
kubectl logs -n operators deploy/my-operator -c manager -f

# Inspect CRD for validation issues
kubectl get crd mydatabase.example.com -o yaml

# Check CR status and conditions
kubectl get mydatabase example -o jsonpath='{.status}'

# View detailed CR description including events
kubectl describe mydatabase example

# Verify RBAC permissions
kubectl auth can-i create pods --as=system:serviceaccount:operators:my-operator

# Check for errors in controller manager
kubectl describe pod -n operators -l control-plane=controller-manager

# Examine Kubernetes events for the namespace
kubectl get events -n operators --sort-by='.lastTimestamp'

# Verify webhook configuration
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

# Check controller leader election (for HA deployments)
kubectl get lease -n operators

# Test with increased logging verbosity
kubectl patch deployment my-operator -n operators --type=json \
  -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--zap-log-level=debug"}]'

Advanced Debugging Techniques

  • Use tools like delve for remote debugging Go operators
  • Add debug endpoints to your operator for on-demand diagnostics
  • Create a debug build with additional instrumentation
  • Implement trace ID propagation for distributed tracing
  • Use tools like Telepresence for local development against remote cluster

Security Considerations

Production Readiness

Before deploying an operator to production, ensure it meets high standards for reliability, maintainability, and operability.

Checklist

  • Comprehensive tests
    • Unit tests for controller logic
    • Integration tests with real Kubernetes API
    • End-to-end tests for full workflows
    • Chaos testing for resilience
    • Upgrade tests between versions
    • Performance tests under load
    • Security scans and penetration testing
  • Version strategy
    • Semantic versioning for operator releases
    • CRD versioning plan
    • Clear deprecation policies
    • Conversion strategy between versions
    • Compatibility matrix documentation
    • Container image tagging strategy
    • Rollback procedures defined
  • Documentation
    • Installation and configuration guide
    • API reference for all CRDs
    • Operational procedures for common tasks
    • Troubleshooting guides
    • Architectural overview
    • Example use cases and configurations
    • Known limitations and workarounds
  • Monitoring integration
    • Prometheus metrics exposed
    • Default Grafana dashboards
    • Alert definitions for critical conditions
    • Health and readiness endpoints
    • Detailed logging strategy
    • Tracing integration
    • Event recording for significant actions
  • Backup/restore procedures
    • Clear backup methodology
    • Documented restore process
    • Disaster recovery testing
    • Point-in-time recovery options
    • Data consistency guarantees
    • Cross-cluster migration procedures
    • Backup validation mechanisms
  • Update strategy
    • In-place upgrade support
    • Canary deployment options
    • Progressive rollout capabilities
    • Feature flags for gradual enabling
    • A/B testing support
    • Blue/green deployment procedures
    • Automatic or manual approval workflows
  • Error handling
    • Graceful degradation under pressure
    • Comprehensive error logging
    • Self-healing mechanisms
    • Circuit breaking for external dependencies
    • Retry strategies with backoff
    • Failure domain isolation
    • Deterministic error reporting
  • Resource constraints
    • Appropriate resource requests and limits
    • Horizontal scaling capability
    • Vertical scaling considerations
    • Performance under resource pressure
    • Graceful handling of resource exhaustion
    • Quality of service guarantees
    • Prioritization of critical operations

A production-ready operator should be treated like any mission-critical application, with proper CI/CD pipelines, change management procedures, and operational runbooks. The ultimate goal is to make the operator itself as reliable and maintainable as the applications it manages.

Operator Hub

Featured Operators from OperatorHub

Some popular operators available on OperatorHub include:

  1. Prometheus Operator - Automated deployment and management of Prometheus monitoring stacks
  2. Elasticsearch Operator - Manages Elasticsearch, Kibana, and APM Server on Kubernetes
  3. etcd Operator - Manages etcd clusters deployed on Kubernetes
  4. MongoDB Community Kubernetes Operator - Automates and manages MongoDB deployments
  5. Strimzi Kafka Operator - Simplifies running Apache Kafka on Kubernetes
  6. Redis Operator - Creates and maintains Redis clusters
  7. PostgreSQL Operator - Manages PostgreSQL clusters
  8. Jaeger Operator - Simplifies deployment of Jaeger tracing infrastructure

Publishing Your Operator

To publish your operator to OperatorHub:

  1. Package your operator using the Operator Framework bundle format
  2. Ensure it meets the required criteria
  3. Submit a pull request to the community-operators repository
  4. Respond to community review feedback
  5. Maintain your operator with regular updates and improvements

By publishing to OperatorHub, you make your operator discoverable to the wider Kubernetes community and benefit from community feedback and contributions.

Edit this page
Autoscaling & Resource Management
Understanding Kubernetes autoscaling mechanisms and resource management strategies
Deployment Strategies
Understanding various Kubernetes deployment strategies and best practices