Welcome to from-docker-to-kubernetes

Content Trust & Image Signing

Learn how to implement Docker Content Trust and image signing for secure software supply chains

H A R S H H A A
@NotHarshhaa

Docker Content Trust and Image Signing

Docker Content Trust (DCT) provides a way to verify the integrity and publisher of container images, enabling secure software supply chains and ensuring that only trusted content runs in your environment. It implements a signing system that allows users to cryptographically verify the authenticity of images before pulling or running them, which is crucial for security-sensitive applications and environments.

DCT creates a chain of trust from image publishers to consumers, ensuring images haven't been tampered with at any point in the distribution process. This mechanism addresses critical security challenges in container deployments by preventing unauthorized or modified container images from running in your infrastructure.

Core Concepts

Docker Content Trust

  • Verifies image integrity through cryptographic hashing of all layers
  • Validates image publisher identity using digital signatures
  • Prevents running tampered images by rejecting unsigned or invalidly signed content
  • Integrates with Docker CLI through native commands and environment variables
  • Provides chain of trust from publisher to runtime environment
  • Enables non-repudiation of image publishing actions
  • Secures the entire software supply chain from build to deployment

Signing System

  • Uses Notary project, an open-source implementation of a signing system
  • Implements The Update Framework (TUF), a security specification for secure content distribution
  • Manages root keys (highest authority, kept offline) and delegation keys (day-to-day signing)
  • Establishes offline root of trust to minimize exposure of critical keys
  • Maintains online repository metadata for signature verification
  • Creates hierarchical trust relationships for flexible signing workflows
  • Includes timestamp services to prevent replay attacks

Enabling Content Trust

Key Management

# Generate and load keys (done automatically on first push with DCT enabled)
docker trust key generate my-signing-key
# This creates a new signing key pair and stores them in your local Docker trust store
# The private key is protected with a passphrase you'll need to provide
# The public key is used for verification and can be shared

# Add a signer to a repository
docker trust signer add --key my-signing-key.pub alice myorg/myimage
# This adds a new authorized signer (alice) to the repository
# Only designated signers can create valid signatures for the repository
# Enables delegation of signing authority to multiple team members
# Requires repository admin permissions

# Remove a signer
docker trust signer remove alice myorg/myimage
# Revokes signing privileges for the specified user
# Previously signed images remain valid
# Future signatures from this signer will be rejected
# Useful when team members leave or responsibilities change

# View signing keys
docker trust key ls
# Lists all signing keys available to the current user
# Shows key IDs, types, and locations
# Helps manage your signing infrastructure
# Does not display private key material

# Rotate keys
docker trust key rotate myorg/myimage
# Updates the signing keys for a repository
# Critical for security after potential compromise
# Establishes new trust chain while maintaining image access
# Should be part of regular security maintenance

Docker creates two types of keys when you first enable DCT:

  • Root key: Your most valuable key that should be kept offline and secure
  • Repository key: Generated for each repository you push to, used for day-to-day signing

Signing Images

Basic Image Signing

# With DCT enabled, signing happens automatically on push
export DOCKER_CONTENT_TRUST=1
docker push myorg/myimage:latest
# Docker will prompt for your signing key passphrase
# Image layers and manifest are cryptographically signed
# Signatures are stored in the Notary server (often co-located with registry)
# The signed content can now be verified by consumers

# Explicitly sign an image that was previously pushed
docker trust sign myorg/myimage:latest
# Signs an existing image without pushing it again
# Useful for retroactively signing images
# Adds your signature to the trust metadata
# Requires the image to already exist in the registry

Viewing Signature Information

# Inspect trust data for an image
docker trust inspect myorg/myimage:latest
# Shows signers, roles, and signature metadata
# Displays signing timestamps and expiration dates
# Lists key IDs used for signing
# JSON format output for programmatic processing

# Inspect with expanded certificates and human-readable format
docker trust inspect --pretty myorg/myimage:latest
# Provides a more readable output format with formatted dates
# Shows detailed certificate information
# Displays hierarchical trust relationships
# Better for human analysis and troubleshooting

When Docker trust is enabled, every image pull operation will verify the signatures against the Notary server before allowing the image to be used, providing continuous protection against image tampering.

Automating Image Signing

# Example GitHub Actions workflow with signing
name: Build and Sign Image

on:
  push:
    branches: [ main ]

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
      
      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      
      - name: Setup Docker Content Trust
        run: |
          mkdir -p ~/.docker/trust
          echo "${{ secrets.DCT_PRIVATE_KEY }}" > ~/.docker/trust/private.key
          echo "${{ secrets.DCT_PASSPHRASE }}" > ~/.docker/trust/passphrase
          export DOCKER_CONTENT_TRUST=1
          export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="$(cat ~/.docker/trust/passphrase)"
          
      - name: Build and Push
        run: |
          docker build -t myorg/myimage:${{ github.sha }} .
          docker tag myorg/myimage:${{ github.sha }} myorg/myimage:latest
          docker push myorg/myimage:${{ github.sha }}
          docker push myorg/myimage:latest

Notary Architecture

Notary Components

  • Notary Server: Stores and serves signed metadata for repositories
    • Acts as a trusted source of signature verification data
    • Provides secure storage for all signature metadata
    • Implements TUF protocol for secure updates
    • Exposes API for clients to fetch and update trust data
  • Notary Signer: Handles signing operations securely
    • Manages private keys for signing operations
    • Provides isolated environment for cryptographic operations
    • Offers hardware security module (HSM) integration
    • Maintains separation of concerns from metadata storage
  • Notary Client: CLI tool for managing content trust
    • Interacts with both server and signer
    • Provides user interface for trust operations
    • Manages local key storage and signing requests
    • Validates and updates trust metadata
  • TUF Metadata: Multiple layers of signed metadata files
    • Implements The Update Framework specification
    • Provides defense-in-depth through hierarchical trust
    • Protects against various attacks including freeze, rollback, and mix-and-match
    • Enables secure metadata updates even over untrusted channels

Trust Metadata

  • Root: Top-level key, typically offline
    • Highest authority in the trust hierarchy
    • Signs and authorizes all other top-level keys
    • Stored offline to maximize security
    • Long expiration time (typically 1+ years)
    • Used rarely, primarily for key rotation
  • Targets: Controls content hashes
    • Contains hashes of the actual container images
    • Maps image tags to their cryptographic digests
    • Signed by targets key or delegation keys
    • Defines the content that is officially approved
    • Can be extended with custom metadata
  • Snapshot: Records current collection of metadata
    • Prevents mix-and-match attacks
    • Contains references to all current metadata versions
    • Ensures consistency across all trust information
    • Updated whenever any metadata changes
    • Provides a consistent view of repository state
  • Timestamp: Indicates freshness of repository
    • Prevents replay attacks with expired metadata
    • Updated frequently (typically daily)
    • Has shortest expiration of all metadata
    • Signed by online keys for automation
    • Confirms the repository is actively maintained
  • Delegations: Additional signing authorities
    • Enables distributed trust model
    • Allows multiple signers for different paths
    • Supports team-based signing workflows
    • Can create hierarchical signing responsibilities
    • Enables fine-grained access control for signing

Advanced Signing with Notary

# Initialize a Notary repository
notary init example.com/myorg/myimage
# Creates a new trusted collection for your image
# Generates initial root, targets, snapshot, and timestamp metadata
# Establishes trust hierarchy for the repository
# You'll need to provide a passphrase for the root and repository keys

# Add a delegation key
notary delegation add example.com/myorg/myimage targets/releases alice.crt --all-paths
# Creates a delegation role called "releases" under the targets role
# Adds Alice's certificate as an authorized signer for this delegation
# The --all-paths flag allows signing any path in the repository
# For more granular control, specify paths like "path:v1*" instead

# Publish changes
notary publish example.com/myorg/myimage
# Uploads the locally modified trust metadata to the Notary server
# Makes your changes visible to other users of the repository
# Updates the snapshot and timestamp metadata with current state
# Requires authentication to the Notary server

# List keys in Notary
notary key list
# Shows all keys in your local Notary trust store
# Displays key IDs, locations, roles, and key types
# Helps inventory your signing capabilities
# Does not show private key material or passphrases

# Rotate a key
notary key rotate example.com/myorg/myimage targets/releases
# Replaces the existing key with a new one for the specified role
# Maintains trust chain during key transition
# Critical for security after suspected compromise
# Updates delegation metadata to use new key
# All previously signed content remains valid

The Notary CLI provides a more direct and granular interface to the trust system than the Docker trust commands, making it valuable for advanced trust management scenarios and automation.

Cosign Integration

Policy Enforcement

Admission Controllers

  • Kubernetes validating webhooks that intercept and validate requests
    • Run as independent services within your Kubernetes cluster
    • Receive admission requests before objects are persisted
    • Can approve, reject, or modify incoming resources
    • Operate asynchronously to minimize performance impact
  • Enforce signature verification at deployment time
    • Check image signatures before allowing pods to run
    • Verify signatures against trusted public keys
    • Support multiple verification providers (DCT, Cosign, etc.)
    • Implement configurable verification policies
  • Block unsigned or untrusted images from running
    • Reject deployments with unverified images
    • Provide detailed rejection reasons in error messages
    • Create exceptions for specific namespaces or sources
    • Support emergency bypass mechanisms with proper authorization
  • Implement organization policies for image trust
    • Define allowed signers for different environments
    • Set minimum signature requirements by namespace
    • Create tiered trust policies (dev, staging, production)
    • Enforce supply chain security requirements
  • Audit and compliance reporting for security governance
    • Log all verification decisions with detailed metadata
    • Generate reports on signature compliance
    • Track policy exceptions and bypass events
    • Provide evidence for security audits and certifications
    • Integrate with security information and event management (SIEM) systems

Example Admission Policy

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: image-signature-webhook
webhooks:
- name: image-verify.example.com
  clientConfig:
    service:
      namespace: image-verify
      name: image-verify-webhook
      path: "/validate"
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  failurePolicy: Fail
  sideEffects: None
  admissionReviewVersions: ["v1"]

Software Supply Chain Security

graph TD
    A[Developer Commits Code] --> B[CI/CD Pipeline]
    B --> C[Build Container Image]
    C --> D[Sign Image]
    D --> E[Push to Registry]
    E --> F[Image Scanning]
    F --> G[Deploy to Kubernetes]
    H[Admission Controller] --> |Verify Signature| G
    I[Key Management] --> D
    J[Security Policies] --> H

Securing Private Registries

Vulnerability Management

Integration with Scanning

  • Sign images after security scanning to certify their security status
    • Run comprehensive vulnerability scans before signing
    • Only sign images that meet security thresholds
    • Include scan results as attestations with signatures
    • Create secure pipeline stages for scan-then-sign workflow
    • Document the security baseline for each signed image
  • Revoke signatures for vulnerable images when new threats emerge
    • Monitor CVE databases for new vulnerabilities
    • Implement automatic signature revocation for critical issues
    • Notify teams when signatures are revoked
    • Provide clear paths for remediation
    • Track vulnerability-to-revocation timeframes
  • Implement re-signing after patching to maintain trusted status
    • Verify patch effectiveness before re-signing
    • Update attestations with new security information
    • Maintain version history of signature changes
    • Document justification for each re-signing event
    • Implement proper approval workflows for re-signing
  • Automated security update workflow for continuous protection
    • Detect vulnerable components automatically
    • Build, test, and sign patched images
    • Deploy updates with minimal human intervention
    • Verify successful update deployment
    • Roll back automatically if verification fails
  • Policy-based signature enforcement tied to security requirements
    • Define security levels required for different environments
    • Implement stricter policies for production workloads
    • Create exception processes for emergency situations
    • Review and update policies based on threat landscape
    • Align policies with regulatory requirements

Continuous Validation

  • Regular verification of deployed images beyond initial deployment
    • Periodic signature validation of running containers
    • Compare running images against trusted signatures
    • Detect container drift or tampering
    • Implement verification as part of health checks
    • Create reconciliation loops for continuous compliance
  • Monitoring for compromised signatures and trust issues
    • Watch for revoked certificates or compromised keys
    • Alert on unexpected signature changes
    • Implement dead man's switch for timestamp validation
    • Track signature expiration to prevent trust lapses
    • Monitor Notary server health and security
  • Automated remediation for policy violations
    • Define clear action plans for different violation types
    • Implement graduated response based on severity
    • Create automated quarantine procedures for suspect containers
    • Build self-healing capabilities for trust violations
    • Document remediation actions for compliance purposes
  • Audit trails for image provenance across the entire lifecycle
    • Record all signature events with complete metadata
    • Track image history from build to retirement
    • Implement immutable logging for signature operations
    • Create verifiable evidence chains for forensic analysis
    • Maintain historical records for compliance requirements
  • Supply chain transparency for all stakeholders
    • Provide visibility into signing authorities and processes
    • Document trust relationships in a human-readable format
    • Create dashboards showing image trust status
    • Enable verification of supply chain integrity
    • Support third-party audits of trust infrastructure

Best Practices

Key Management

  • Store root keys offline (hardware tokens)
    • Use hardware security modules (HSMs) or smart cards
    • Store root keys in air-gapped systems
    • Implement physical security controls for offline keys
    • Consider tamper-evident storage for critical keys
    • Create ceremonies for root key operations
  • Use separate keys for different environments
    • Maintain strict separation between development and production keys
    • Create environment-specific trust hierarchies
    • Implement different security levels for different key types
    • Use separate delegations for different teams or services
    • Never share keys between security boundaries
  • Implement secure key rotation procedures
    • Schedule regular rotation based on key sensitivity
    • Create clear procedures for planned and emergency rotations
    • Test rotation procedures before implementation
    • Maintain backward compatibility during transition periods
    • Document each rotation event for audit purposes
  • Define clear key custodian responsibilities
    • Assign specific roles for key management
    • Implement separation of duties for critical operations
    • Create backup custodian assignments
    • Train custodians on security procedures
    • Implement multi-person control for highest-value keys
  • Document key recovery processes
    • Create detailed recovery procedures for key loss scenarios
    • Test recovery processes regularly
    • Store recovery materials securely with appropriate access controls
    • Implement break-glass procedures for emergency access
    • Consider legal and compliance implications of recovery

Operational Security

  • Enforce content trust in production environments
    • Make signature verification mandatory for production workloads
    • Implement technical controls to prevent bypass
    • Create clear exception processes for emergency scenarios
    • Regularly test enforcement mechanisms
    • Document compliance with trust requirements
  • Train developers on signing workflow and security implications
    • Create educational materials on content trust concepts
    • Provide hands-on training for signing procedures
    • Include content trust in security awareness programs
    • Train incident responders on trust-related issues
    • Make security a shared responsibility across teams
  • Automate signing in CI/CD pipelines for consistency
    • Design secure signing stages in build pipelines
    • Implement proper secret management for signing keys
    • Create approval gates before signing operations
    • Log and monitor all automated signing activities
    • Test signing automation regularly
  • Regularly audit signature policies and implementation
    • Conduct periodic reviews of trust configuration
    • Verify adherence to signing policies
    • Test signature verification on random samples
    • Update policies based on emerging threats
    • Include trust systems in security assessments
  • Monitor for unauthorized image usage or trust violations
    • Implement real-time alerting for trust failures
    • Create dashboards for trust status across environments
    • Use anomaly detection for unusual signing patterns
    • Track attempted uses of unsigned images
    • Include trust violations in security incident procedures

Troubleshooting

Modern Approaches and Tools

Sigstore and Cosign

  • Keyless signing options that eliminate private key management
    • Uses short-lived certificates tied to identity
    • Leverages OpenID Connect (OIDC) for authentication
    • Provides non-repudiation without key storage concerns
    • Issues ephemeral certificates from Fulcio CA
    • Minimizes risk of key compromise
  • Transparency logs for public verification and auditability
    • Records all signatures in a tamper-evident log (Rekor)
    • Enables third-party verification of signing events
    • Creates immutable audit trails of signature activities
    • Supports inclusion proofs for signature verification
    • Makes supply chain activities publicly verifiable
  • OIDC identity integration for human and workload authentication
    • Uses existing identity providers (GitHub, Google, etc.)
    • Supports both human and service account identities
    • Ties signatures directly to authenticated identities
    • Enables identity-based verification policies
    • Simplifies access management for signing operations
  • Simplified developer experience with modern tooling
    • Reduces complexity compared to traditional PKI
    • Provides clear, user-friendly commands and interfaces
    • Integrates with existing development workflows
    • Supports multiple signature formats and verification methods
    • Reduces friction in adopting supply chain security
  • Kubernetes integration through admission controllers and policies
    • Works with Policy Controller and OPA Gatekeeper
    • Provides CRDs for Kubernetes-native policy definition
    • Integrates with Kubernetes RBAC for authorization
    • Supports namespace-level policy configuration
    • Enables cluster-wide signature enforcement

In-toto Attestations

  • Capture entire build provenance with detailed metadata
    • Records all steps in the build and release process
    • Documents inputs, outputs, and environment for each step
    • Creates verifiable evidence of build process integrity
    • Supports reproducible builds verification
    • Enables complete software bill of materials generation
  • Link steps in build process with cryptographic guarantees
    • Creates a verifiable chain of operations
    • Ensures each step was performed as expected
    • Detects tampering or unauthorized modifications
    • Provides transitive trust across the pipeline
    • Enforces order and dependencies between steps
  • Verify complete supply chain from source to deployment
    • Validates every transformation in the software lifecycle
    • Ensures no unauthorized steps were introduced
    • Verifies all components meet security requirements
    • Creates end-to-end verification of software provenance
    • Supports complex supply chain topologies
  • Additional metadata beyond signatures for rich verification
    • Includes vulnerability scan results
    • Documents policy compliance status
    • Records build environment configuration
    • Captures test results and quality metrics
    • Provides licensing and compliance information
  • Framework for custom attestations tailored to specific needs
    • Supports domain-specific attestation types
    • Enables creation of custom verification rules
    • Allows extension for organization-specific requirements
    • Provides standardized format for diverse metadata
    • Integrates with existing security and compliance systems

Future Directions

Edit this page
Docker Contexts
Learn how to manage multiple Docker environments with Docker contexts for seamless switching between different Docker endpoints
Docker for AI/ML Workloads
Learn how to effectively containerize, deploy, and orchestrate AI and machine learning workloads with Docker