Monitoring & Logging

Built-in Monitoring

Docker provides basic monitoring capabilities out of the box that require no additional tools or setup:

# View real-time resource usage for all containers
docker stats
# Sample output:
# CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
# 1a2b3c4d5e6f   web       0.25%     15.23MiB / 1.952GiB   0.76%     648B / 648B       0B / 0B           2
# 2a3b4c5d6e7f   redis     0.05%     2.746MiB / 1.952GiB   0.14%     1.45kB / 1.05kB   0B / 0B           5

# View detailed container stats in JSON format (useful for scripting)
docker stats --no-stream --format "{{json .}}" container_name | jq '.'
# Pretty prints JSON output with jq for better readability

# Filter stats to show only specific containers
docker stats $(docker ps --format {{.Names}} | grep "api-")

# Display specific columns only
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# Inspect container configuration details
docker inspect container_name

# Extract specific configuration information
docker inspect --format '{{.HostConfig.Memory}}' container_name
docker inspect --format '{{.State.Health.Status}}' container_name
docker inspect --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name

# View container events in real-time
docker events --filter 'type=container'

Built-in monitoring tools provide a good starting point but have limitations:

• No historical data retention
• Limited metrics granularity
• No alerting capabilities
• Minimal visualization options
• Container-centric rather than application-centric view

cAdvisor

Google's container advisor (cAdvisor) provides more detailed container metrics with a web UI and exposes Prometheus-compatible endpoints:

# Run cAdvisor container with all necessary volume mounts
docker run -d \
  --name=cadvisor \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --privileged \
  --device=/dev/kmsg \
  --restart=always \
  gcr.io/cadvisor/cadvisor:v0.47.0

cAdvisor provides:

• Detailed resource usage statistics (CPU, memory, network, filesystem)
• Container lifecycle events
• Historical performance data (short-term, in-memory storage)
• Container hierarchy visualization
• Prometheus metrics endpoint at /metrics
• Hardware-specific monitoring (NVML for NVIDIA GPUs)
• Auto-discovery of containers

Key metrics exposed by cAdvisor:

• CPU usage breakdown (user, system, throttling)
• Memory details (RSS, cache, swap, working set)
• Network statistics (RX/TX bytes, errors, dropped packets)
• Filesystem usage and I/O statistics
• Per-process statistics within containers

Accessing the UI:

• Web interface at http://localhost:8080
• API endpoints for programmatic access:
  • /api/v1.3/containers - All container stats
  • /api/v1.3/docker/[container_name] - Specific container stats
  • /metrics - Prometheus-formatted metrics

Prometheus + Grafana

A powerful combination for metrics collection, storage, querying, and visualization, widely considered the industry standard for container monitoring:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.44.0
    container_name: prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=15d'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    restart: unless-stopped
    
  node-exporter:
    image: prom/node-exporter:v1.5.0
    container_name: node-exporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - "9100:9100"
    restart: unless-stopped
    
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    restart: unless-stopped
    
  grafana:
    image: grafana/grafana:9.5.1
    container_name: grafana
    user: "472"
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin_password
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_SERVER_DOMAIN=localhost
    restart: unless-stopped

volumes:
  prometheus_data: {}
  grafana_data: {}

Example prometheus.yml configuration:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
    
  - job_name: 'docker'
    metrics_path: /metrics
    static_configs:
      - targets: ['172.17.0.1:9323']  # Docker daemon metrics (requires daemon configuration)

  # Auto-discover and scrape containers with prometheus.io annotations
  - job_name: 'docker-containers'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        filters:
          - name: label
            values: ['prometheus.io.scrape=true']
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_prometheus_io_port']
        target_label: __metrics_path__
        replacement: /metrics
      - source_labels: ['__meta_docker_container_name']
        target_label: container_name
      - source_labels: ['__meta_docker_container_label_prometheus_io_job']
        target_label: job

This setup provides:

• Prometheus: Time-series database with a powerful query language (PromQL)
• Node Exporter: Host-level metrics (CPU, memory, disk, network)
• cAdvisor: Container-specific metrics
• Grafana: Visualization dashboards and alerting

You can enable Docker daemon metrics by adding to /etc/docker/daemon.json:

{
  "metrics-addr": "0.0.0.0:9323",
  "experimental": true
}

Common Prometheus metrics for containers:

• container_cpu_usage_seconds_total - Cumulative CPU time consumed
• container_memory_usage_bytes - Current memory usage, including cache
• container_memory_rss - Resident Set Size (actual memory usage)
• container_network_receive_bytes_total - Network bytes received
• container_network_transmit_bytes_total - Network bytes sent
• container_fs_reads_bytes_total - Bytes read from disk
• container_fs_writes_bytes_total - Bytes written to disk

Example PromQL queries:

# CPU usage percentage per container
sum(rate(container_cpu_usage_seconds_total{name=~".+"}[1m])) by (name) * 100

# Memory usage in MB per container
sum(container_memory_usage_bytes{name=~".+"}) by (name) / 1024 / 1024

# Network receive throughput
rate(container_network_receive_bytes_total{name=~".+"}[5m])

Basic Log Commands

# View container logs (shows all logs)
docker logs container_name

# Follow container logs in real-time (like tail -f)
docker logs -f container_name

# Show timestamps with logs (useful for debugging)
docker logs -t container_name

# Show last n lines (instead of entire log history)
docker logs --tail=100 container_name

# Combine options for real-time monitoring with timestamps
docker logs -f -t --tail=50 container_name

# Filter logs by time (show logs since specific timestamp)
docker logs --since 2023-07-01T00:00:00 container_name

# Show logs until a specific timestamp
docker logs --until 2023-07-02T00:00:00 container_name

# Show logs from relative time (e.g., last hour)
docker logs --since=1h container_name

# Grep for specific patterns in logs
docker logs container_name | grep ERROR

# Output logs in different formats (useful for parsing)
docker logs --details container_name

# Count occurrences of specific log patterns
docker logs container_name | grep -c "Connection refused"

# Extract and format log timestamps for timing analysis
docker logs -t container_name | awk '{print $1, $2}' | sort

# Monitor multiple containers simultaneously
docker logs $(docker ps -q) 2>&1 | grep ERROR

# Show container logs with colorized output for different log levels
docker logs container_name | grep --color -E "ERROR|WARN|INFO|DEBUG|$"

Note that docker logs only works with containers using the json-file, local, or journald logging drivers. For other logging drivers (like syslog, fluentd, etc.), you'll need to access logs through their respective systems.

Log Retrieval Performance Considerations

• For large log volumes, use --tail to limit output
• Avoid frequent calls to docker logs on busy production systems
• Consider using the --since flag to limit time range
• For high-traffic containers, use a dedicated logging solution instead of docker logs
• The docker logs command reads the entire log file before applying filters, which can be resource-intensive

Log Rotation

Log rotation is essential to prevent disk space exhaustion. Configure it in the Docker daemon settings:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",      // Maximum size of each log file before rotation
    "max-file": "3",        // Number of log files to keep
    "compress": "true"      // Compress rotated log files
  }
}

Container-specific log rotation can be configured at runtime:

# Configure log rotation for a specific container
docker run -d \
  --log-opt max-size=5m \
  --log-opt max-file=5 \
  --log-opt compress=true \
  --name web nginx

For existing deployments with unmanaged log files, you can use external log rotation:

# Example logrotate configuration (/etc/logrotate.d/docker-containers)
/var/lib/docker/containers/*/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

You can manually trigger log rotation by restarting the container or the Docker daemon.

Common log rotation issues:

• Docker daemon restart required for daemon.json changes to take effect
• Containers created before log rotation configuration won't have it applied
• Very high log volume containers may still face issues despite rotation
• copytruncate in logrotate may miss logs during rotation window
• Nested JSON logs can be challenging to parse after rotation

Multi-container Log Aggregation

Tools like Fluentd, Logstash, or Filebeat can collect logs from multiple containers and forward them to centralized logging systems:

version: '3.8'
services:
  app:
    image: my-app
    logging:
      driver: fluentd
      options:
        fluentd-address: fluentd:24224
        tag: app.{{.Name}}
        fluentd-async: "true"
        fluentd-async-connect: "true"
        labels: "environment,service_name,version"
        env: "NODE_ENV,SERVICE_VERSION"
    labels:
      environment: production
      service_name: api
      version: "1.0.0"
    environment:
      NODE_ENV: production
      SERVICE_VERSION: "1.0.0"
  
  web:
    image: nginx
    logging:
      driver: fluentd
      options:
        fluentd-address: fluentd:24224
        tag: web.{{.Name}}
        fluentd-async: "true"
    depends_on:
      - app
  
  db:
    image: postgres
    logging:
      driver: fluentd
      options:
        fluentd-address: fluentd:24224
        tag: db.{{.Name}}
        fluentd-async: "true"
  
  fluentd:
    image: fluentd/fluentd:v1.16-1
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - fluentd-data:/fluentd/log
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    environment:
      - FLUENTD_CONF=fluent.conf
    restart: always

volumes:
  fluentd-data:

Example Fluentd configuration (fluent.conf):

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

# Parse Docker logs
<filter **>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
    json_parser json
  </parse>
</filter>

# Add Kubernetes metadata if running in K8s
<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

# Output to Elasticsearch
<match **>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix fluentd
  <buffer>
    @type file
    path /fluentd/log/buffer
    flush_thread_count 2
    flush_interval 5s
    chunk_limit_size 2M
    queue_limit_length 32
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

# Keep a local copy of logs for debugging
<match **>
  @type copy
  <store>
    @type file
    path /fluentd/log/${tag}/%Y/%m/%d.%H.%M
    append true
    <format>
      @type json
    </format>
    <buffer tag,time>
      @type file
      timekey 1h
      timekey_use_utc true
      timekey_wait 10m
    </buffer>
  </store>
</match>

Alternative log aggregation solutions:

• Filebeat: Lightweight log shipper from Elastic Stack
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.8.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    user: root
    restart: always
  

• Logstash: More powerful log processing pipeline
  logstash:
    image: docker.elastic.co/logstash/logstash:8.8.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx512m -Xms512m"
    restart: always
  

• Vector: High-performance observability data pipeline
  vector:
    image: timberio/vector:0.29.1-alpine
    volumes:
      - ./vector.toml:/etc/vector/vector.toml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "8686:8686"
    restart: always
  

Basic Setup

1. Access Grafana (default: http://localhost:3000)
   • Ensure Grafana service is running and port is accessible
   • Check for any proxy or network restrictions
2. Login with default credentials (admin/admin)
   • You'll be prompted to change the password on first login
   • Set a strong password and store it securely
   • Consider setting up additional users with appropriate permissions
3. Add Prometheus data source
   • Navigate to Configuration > Data Sources
   • Click "Add data source" and select "Prometheus"
   • Set URL to http://prometheus:9090 (or appropriate address)
   • Set scrape interval to match Prometheus configuration
   • Test the connection to ensure it works
   • Advanced settings:
     HTTP Method: GET
     Type: Server (default)
     Access: Server (default)
     Disable metrics lookup: No
     Custom query parameters: None
     

4. Import Docker monitoring dashboards
   • Navigate to Dashboards > Import
   • Enter dashboard ID or upload JSON file
   • Recommended dashboard IDs:
     • 893: Docker and system monitoring (1 server)
     • 10619: Docker monitoring with Prometheus
     • 11467: Container metrics dashboard
     • 1860: Node Exporter Full
   • Adjust variables to match your environment
   • Save dashboard with appropriate name and folder
5. Configure alerts
   • Navigate to Alerting > Alert Rules
   • Create alert rules based on critical metrics
   • Set appropriate thresholds and evaluation intervals
   • Configure notification channels (email, Slack, PagerDuty, etc.)
   • Test alerts to ensure proper delivery
   • Example alert rule:
     Rule name: High Container CPU Usage
     Data source: Prometheus
     Expression: max by(container_name) (rate(container_cpu_usage_seconds_total{container_name!=""}[1m]) * 100) > 80
     Evaluation interval: 1m
     Pending period: 5m
     

Dashboard Recommendations

• Docker Host metrics dashboard
  • System load, CPU, memory, disk, and network
  • Host uptime and stability metrics
  • Docker daemon metrics
  • Number of running containers
  • Resource utilization trends
  • Example panels:
    • CPU Usage by Container (stacked graph)
    • Memory Usage by Container (stacked graph)
    • Container Status Count (stat panel)
    • System Load (gauge)
    • Disk Space Usage (pie chart)
• Container resource usage dashboard
  • Per-container CPU, memory, and I/O metrics
  • Container restart counts
  • Health check status
  • Network traffic by container
  • Key visualizations:
    • Heatmap of container resource usage
    • Time-series charts for each resource type
    • Top N resource consumers (table)
    • Container lifecycle events (annotations)
    • Resource limit vs. actual usage comparison
• Application-specific metrics dashboards
  • Business KPIs relevant to your application
  • Request rates, error rates, and latencies
  • Database connection pool status
  • Cache hit ratios
  • Custom instrumentation metrics
  • User experience metrics
  • Example: E-commerce dashboard with:
    • Orders per minute
    • Cart abandonment rate
    • Payment processing time
    • Product search latency
    • Active user sessions
• Alert thresholds visualization
  • Combine metrics with alert thresholds
  • Visual indicators for approaching thresholds
  • Alert history and frequency
  • Mean time to resolution tracking
  • Alert correlation with system events
  • Example panel: Graph with colored threshold bands
• Log correlation views
  • Combined metrics and log panels
  • Error rate correlation with log volume
  • Event markers on time-series charts
  • Log context for anomalies
  • Drill-down from metrics to logs
  • Example: Request latency chart with error log entries as annotations

Distributed Tracing

• Implement OpenTelemetry or Jaeger
  • Track requests as they flow through distributed systems
  • Generate trace IDs to correlate logs across services
  • Instrument code with OpenTelemetry SDK
  • Deploy Jaeger or Zipkin collectors
  • Example Jaeger deployment:
    version: '3.8'
    services:
      jaeger:
        image: jaegertracing/all-in-one:1.37
        ports:
          - "6831:6831/udp"  # Jaeger agent - accepts spans in Thrift format
          - "16686:16686"    # Jaeger UI
        environment:
          - COLLECTOR_ZIPKIN_HOST_PORT=:9411
    

• Trace requests across multiple containers
  • Propagate trace context between services
  • Capture request path across microservices
  • Record parent-child relationships between spans
  • Preserve baggage items for request context
  • Example client instrumentation (Node.js):
    const tracer = opentelemetry.trace.getTracer('my-service');
    const span = tracer.startSpan('process-order');
    try {
      // Add attributes to span
      span.setAttribute('order.id', orderId);
      span.setAttribute('customer.id', customerId);
      
      // Create child span
      const dbSpan = tracer.startSpan('database-query', {
        parent: span,
      });
      // Process database operations
      dbSpan.end();
      
      // Propagate context to HTTP requests
      const headers = {};
      opentelemetry.propagation.inject(opentelemetry.context.active(), headers);
      await axios.post('http://inventory-service/check', data, { headers });
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR });
    } finally {
      span.end();
    }
    

• Measure service latency
  • Calculate time spent in each service
  • Break down processing time by operation
  • Identify slow components or dependencies
  • Compare latency across different environments
  • Correlate latency with resource utilization
  • Example span attributes for latency analysis:
    database.query.duration_ms: 45.2
    http.request.duration_ms: 120.7
    cache.lookup_time_ms: 2.1
    business_logic.processing_time_ms: 15.8
    

• Identify bottlenecks
  • Analyze critical path in request processing
  • Find operations with highest latency contribution
  • Detect contention points and resource constraints
  • Quantify impact of external dependencies
  • Trace-based hotspot analysis techniques
  • Example hotspot dashboard showing service latency breakdown
• Visualize service dependencies
  • Generate service dependency graphs
  • Analyze traffic patterns between services
  • Calculate error rates between service pairs
  • Identify redundant or unnecessary calls
  • Detect circular dependencies
  • Example visualization tools:
    • Jaeger UI service graph
    • Grafana service graph panel
    • Kiali for service mesh visualization
    • Custom D3.js visualization

Custom Metrics

• Expose application-specific metrics
  • Identify key business and technical metrics
  • Instrument application code with metrics
  • Expose metrics endpoints (/metrics)
  • Design meaningful metric names and labels
  • Balance cardinality and usefulness
  • Example custom metrics:
    # Counter for business events
    order_total{status="completed",payment_method="credit_card"} 1550.75
    
    # Gauge for active resource usage
    active_user_sessions{region="us-west"} 1250
    
    # Histogram for latency distributions
    api_request_duration_seconds_bucket{endpoint="/api/v1/products",le="0.1"} 1500
    api_request_duration_seconds_bucket{endpoint="/api/v1/products",le="0.5"} 1950
    api_request_duration_seconds_bucket{endpoint="/api/v1/products",le="1.0"} 1990
    api_request_duration_seconds_bucket{endpoint="/api/v1/products",le="+Inf"} 2000
    

• Implement Prometheus client libraries
  • Use official client libraries for language-specific instrumentation
  • Create counters, gauges, histograms, and summaries
  • Register metrics with registry
  • Set up middleware for standard metrics
  • Add custom labels for filtering and aggregation
  • Example implementation (Python):
    from prometheus_client import Counter, Gauge, Histogram, Summary, start_http_server
    
    # Create metrics
    REQUEST_COUNT = Counter('app_request_count', 'Application Request Count', ['method', 'endpoint', 'status'])
    REQUEST_LATENCY = Histogram('app_request_latency_seconds', 'Application Request Latency', ['method', 'endpoint'])
    ACTIVE_SESSIONS = Gauge('app_active_sessions', 'Active Sessions', ['region'])
    
    # Start metrics endpoint
    start_http_server(8000)
    
    # Update metrics in code
    def process_request(request):
        ACTIVE_SESSIONS.labels(region='us-west').inc()
        
        with REQUEST_LATENCY.labels(method='POST', endpoint='/api/v1/order').time():
            # Process request
            result = handle_request(request)
        
        REQUEST_COUNT.labels(method='POST', endpoint='/api/v1/order', status=result.status_code).inc()
        ACTIVE_SESSIONS.labels(region='us-west').dec()
        
        return result
    

• Create custom dashboards
  • Design purpose-specific visualization panels
  • Combine technical and business metrics
  • Create drill-down capabilities
  • Use appropriate visualization types
  • Implement dynamic variables for filtering
  • Example Grafana dashboard JSON structure:
    {
      "title": "E-Commerce Platform Dashboard",
      "panels": [
        {
          "title": "Order Volume by Payment Method",
          "type": "barchart",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "sum by(payment_method) (order_total)",
              "legendFormat": "{{payment_method}}"
            }
          ]
        },
        {
          "title": "API Latency (95th Percentile)",
          "type": "timeseries",
          "datasource": "Prometheus",
          "targets": [
            {
              "expr": "histogram_quantile(0.95, sum(rate(api_request_duration_seconds_bucket[5m])) by (le, endpoint))",
              "legendFormat": "{{endpoint}}"
            }
          ]
        }
      ]
    }
    

• Set relevant thresholds
  • Define SLIs (Service Level Indicators) and SLOs (Service Level Objectives)
  • Create alert thresholds based on business impact
  • Implement multi-level thresholds (warning, critical)
  • Use historical data to establish baselines
  • Account for traffic patterns and seasonality
  • Example alert thresholds:
    - alert: APIHighLatency
      expr: histogram_quantile(0.95, sum(rate(api_request_duration_seconds_bucket[5m])) by (le, endpoint)) > 0.5
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High API latency on {{ $labels.endpoint }}"
        description: "95th percentile latency is {{ $value }}s, which exceeds the SLO of 0.5s"
    

• Correlate with business metrics
  • Connect technical metrics to business outcomes
  • Measure conversion impact of performance issues
  • Track costs associated with resource usage
  • Create composite KPI dashboards
  • Implement business impact scoring
  • Example correlation queries:
    # Conversion rate vs. page load time
    sum(rate(purchase_completed_total[1h])) / sum(rate(product_page_view_total[1h]))
    
    # Revenue impact of errors
    sum(rate(order_total{status="error"}[1h]))
    
    # Customer satisfaction correlation
    rate(support_ticket_created{category="performance"}[1d]) / rate(active_user_sessions[1d])
    

Automated Responses

• Implement auto-scaling based on metrics
  • Configure horizontal pod autoscaling
  • Use custom metrics for scaling decisions
  • Set appropriate cooldown periods
  • Implement predictive scaling for known patterns
  • Test scaling behavior under various conditions
  • Example Docker Swarm service scaling:
    # Autoscaling with docker service update
    while true; do
      # Get current metrics
      REQUESTS=$(curl -s http://prometheus:9090/api/v1/query?query=sum\(rate\(http_requests_total\[1m\]\)\) | jq -r '.data.result[0].value[1]')
      
      # Calculate desired replicas (1 replica per 100 requests/second)
      DESIRED=$(echo "$REQUESTS / 100" | bc)
      if [ $DESIRED -lt 2 ]; then DESIRED=2; fi
      if [ $DESIRED -gt 10 ]; then DESIRED=10; fi
      
      # Get current replicas
      CURRENT=$(docker service ls --filter name=myapp --format "{{.Replicas}}" | cut -d '/' -f1)
      
      # Scale if needed
      if [ $DESIRED -ne $CURRENT ]; then
        echo "Scaling from $CURRENT to $DESIRED replicas"
        docker service update --replicas $DESIRED myapp
      fi
      
      sleep 30
    done
    

• Configure auto-healing for failed containers
  • Set appropriate restart policies
  • Implement health checks for accurate failure detection
  • Configure liveness and readiness probes
  • Capture diagnostic information before restarts
  • Implement circuit breakers for dependent services
  • Example Docker Compose configuration:
    services:
      app:
        image: myapp:latest
        restart: unless-stopped
        healthcheck:
          test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
          interval: 10s
          timeout: 5s
          retries: 3
          start_period: 40s
        deploy:
          restart_policy:
            condition: on-failure
            max_attempts: 3
            window: 120s
        logging:
          driver: "json-file"
          options:
            max-size: "10m"
            max-file: "3"
    

• Create runbooks for common issues
  • Document standard troubleshooting procedures
  • Include diagnostic commands and expected output
  • Link alerts to specific runbook sections
  • Provide escalation paths for unresolved issues
  • Maintain version control for runbooks
  • Example runbook structure:
    # API Service High Latency Runbook
    
    ## Symptoms
    - API response time > 500ms (95th percentile)
    - Increased error rate in downstream services
    - Alert: APIHighLatency firing
    
    ## Diagnostic Steps
    1. Check container resource usage:
    

    docker stats api-service
    
    2. Examine database connection pool:
    

    curl http://api-service:8080/actuator/metrics/hikaricp.connections.usage
    
    3. Check for slow queries:
    

    docker logs api-service | grep "slow query"
    
    ## Resolution Steps
    1. If connection pool utilization > 80%:
    - Increase pool size in config
    - Restart service with `docker restart api-service`
    
    2. If slow queries detected:
    - Check database indexes
    - Optimize identified queries
    
    3. If CPU/memory usage high:
    - Scale service: `docker service scale api-service=3`
    
    ## Escalation
    If issue persists after 15 minutes:
    - Contact: database-team@example.com
    - Slack: #database-support
    

• Develop automated remediation
  • Implement scripted responses to common problems
  • Create self-healing capabilities
  • Add circuit breakers for degraded dependencies
  • Implement graceful degradation modes
  • Balance automation with human oversight
  • Example automated remediation script:
    #!/bin/bash
    # Automated database connection reset
    
    # Check connection errors
    ERROR_COUNT=$(docker logs --since 5m db-service | grep "connection reset" | wc -l)
    
    if [ $ERROR_COUNT -gt 10 ]; then
      echo "Detected database connection issues, performing remediation"
      
      # Capture diagnostics before remediation
      docker logs --since 15m db-service > /var/log/remediation/db-$(date +%s).log
      
      # Perform remediation
      docker exec db-service /scripts/connection-reset.sh
      
      # Verify fix
      sleep 5
      NEW_ERRORS=$(docker logs --since 1m db-service | grep "connection reset" | wc -l)
      
      # Notify on outcome
      if [ $NEW_ERRORS -eq 0 ]; then
        echo "Remediation successful" | slack-notify "#monitoring"
      else
        echo "Remediation failed, escalating" | slack-notify "#monitoring" --urgent
        # Trigger PagerDuty incident
        pagerduty-trigger "Database connection issues persist after remediation"
      fi
    fi
    

• Setup escalation policies
  • Define clear escalation thresholds
  • Create tiered response teams
  • Implement on-call rotation schedules
  • Track mean time to acknowledge and resolve
  • Document communication protocols
  • Example escalation policy:
    # PagerDuty escalation policy
    escalation_policies:
      - name: "API Service Escalation"
        description: "Escalation policy for API service incidents"
        num_loops: 2
        escalation_rules:
          - escalation_delay_in_minutes: 15
            targets:
              - id: "PXXXXXX"  # Primary on-call engineer
                type: "user_reference"
          - escalation_delay_in_minutes: 15
            targets:
              - id: "PXXXXXX"  # Secondary on-call engineer
                type: "user_reference"
          - escalation_delay_in_minutes: 30
            targets:
              - id: "PXXXXXX"  # Engineering manager
                type: "user_reference"
    

Logging Best Practices

• Log to STDOUT/STDERR
  • Follow container best practices
  • Enable centralized collection
  • Avoid managing log files inside containers
  • Let Docker logging drivers handle transport
  • Example: Configure applications to write directly to stdout/stderr rather than log files
• Use structured logging format
  • Implement JSON-formatted logs
  • Include consistent metadata fields
  • Use proper data types within JSON
  • Add correlation IDs for request tracking
  • Maintain schema consistency
  • Example structured log:
    {
      "timestamp": "2023-07-10T15:23:45.123Z",
      "level": "error",
      "service": "order-service",
      "message": "Failed to process payment",
      "traceId": "abc123def456",
      "orderId": "order-789",
      "errorCode": "PAYMENT_DECLINED",
      "duration_ms": 345
    }
    

• Implement log rotation
  • Configure size and time-based rotation
  • Set appropriate retention periods
  • Compress rotated logs
  • Monitor disk usage
  • Handle rotation gracefully
  • Example Docker logging configuration:
    {
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "20m",
        "max-file": "5",
        "compress": "true"
      }
    }
    

• Set appropriate log levels
  • Use DEBUG for development environments
  • Use INFO or WARN for production
  • Reserve ERROR for actionable issues
  • Make log levels configurable at runtime
  • Use consistent level definitions across services
  • Example log level configuration:
    logging:
      level:
        root: WARN
        com.example.api: INFO
        com.example.database: WARN
        com.example.payment: INFO
    

• Configure centralized logging
  • Aggregate logs from all containers
  • Implement proper indexing and search
  • Set up log parsing and normalization
  • Configure access controls for logs
  • Establish retention and archival policies
  • Example EFK stack setup:
    • Filebeat or Fluentd to collect logs
    • Elasticsearch for storage and indexing
    • Kibana for visualization and search
    • Curator for index lifecycle management
• Avoid sensitive data in logs
  • Implement data masking for PII
  • Never log credentials or secrets
  • Truncate potentially large payloads
  • Remove sensitive headers
  • Anonymize personal identifiers
  • Example masking implementation:
    function logRequest(req) {
      const sanitized = {
        method: req.method,
        path: req.path,
        query: sanitizeObject(req.query),
        headers: sanitizeHeaders(req.headers),
        body: sanitizeObject(req.body)
      };
      logger.info({ request: sanitized }, "Incoming request");
    }
    
    function sanitizeObject(obj) {
      const masked = {...obj};
      const sensitiveFields = ['password', 'token', 'ssn', 'credit_card'];
      
      for (const field of sensitiveFields) {
        if (masked[field]) masked[field] = '***REDACTED***';
      }
      return masked;
    }
    

Monitoring Best Practices

• Monitor both host and containers
  • Track host-level resources (CPU, memory, disk, network)
  • Monitor container-specific metrics
  • Correlate container performance with host constraints
  • Watch for noisy neighbor problems
  • Track Docker daemon health
  • Example monitoring stack:
    • Node Exporter for host metrics
    • cAdvisor for container metrics
    • Docker daemon metrics endpoint
    • Process-specific metrics from applications
• Implement alerting with appropriate thresholds
  • Create multi-level alerts (warning/critical)
  • Avoid alert fatigue with proper thresholds
  • Include runbook links in alert notifications
  • Group related alerts to reduce noise
  • Implement alert de-duplication
  • Example alerting configuration:
    - alert: ContainerHighCpuUsage
      expr: rate(container_cpu_usage_seconds_total{name!=""}[1m]) * 100 > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Container CPU usage high ({{ $labels.name }})"
        description: "Container CPU usage is above 80% for 5 minutes"
        runbook_url: "https://wiki.example.com/runbooks/container-cpu"
    

• Use visualization dashboards
  • Create role-specific dashboards
  • Include both overview and detailed views
  • Use appropriate visualization types
  • Implement template variables for filtering
  • Balance information density with readability
  • Example dashboard organization:
    • Executive overview: High-level health and KPIs
    • Operations dashboard: System health and resource usage
    • Developer dashboard: Application performance and errors
    • Service-specific dashboards: Detailed metrics for each service
    • On-call dashboard: Current alerts and recent incidents
• Track business-relevant metrics
  • Monitor key performance indicators (KPIs)
  • Create business-technical correlation views
  • Measure user experience metrics
  • Track conversion and engagement metrics
  • Monitor transaction value and volume
  • Example business metrics:
    # Business metrics in Prometheus format
    # HELP order_value_total Total value of orders in currency
    # TYPE order_value_total counter
    order_value_total{currency="USD",status="completed"} 15234.50
    
    # HELP checkout_started Total number of checkout processes started
    # TYPE checkout_started counter
    checkout_started 5423
    
    # HELP checkout_completed Total number of checkout processes completed
    # TYPE checkout_completed counter
    checkout_completed 4231
    

• Implement health checks
  • Create meaningful application health endpoints
  • Check dependencies in health probes
  • Implement readiness vs. liveness separation
  • Make health checks lightweight
  • Include version and dependency information
  • Example health check endpoint:
    app.get('/health', async (req, res) => {
      try {
        // Check database connectivity
        const dbStatus = await checkDatabase();
        
        // Check cache service
        const cacheStatus = await checkRedis();
        
        // Check external API dependency
        const apiStatus = await checkExternalApi();
        
        const allHealthy = dbStatus && cacheStatus && apiStatus;
        
        res.status(allHealthy ? 200 : 503).json({
          status: allHealthy ? 'healthy' : 'unhealthy',
          version: '1.2.3',
          timestamp: new Date().toISOString(),
          components: {
            database: dbStatus ? 'up' : 'down',
            cache: cacheStatus ? 'up' : 'down',
            api: apiStatus ? 'up' : 'down'
          }
        });
      } catch (error) {
        res.status(500).json({ status: 'error', error: error.message });
      }
    });
    

• Plan for monitoring scalability
  • Design for growth in container count
  • Implement metric aggregation for high-cardinality data
  • Use appropriate retention policies
  • Consider resource requirements for monitoring tools
  • Implement federation for large-scale deployments
  • Example scalability techniques:
    • Prometheus hierarchical federation
    • Service discovery for dynamic environments
    • Metric aggregation and downsampling
    • Sharding metrics by service or namespace
    • Custom recording rules for common queries
    # Recording rules for efficient querying
    groups:
    - name: container_aggregation
      interval: 1m
      rules:
      - record: job:container_cpu:usage_rate5m
        expr: sum(rate(container_cpu_usage_seconds_total[5m])) by (job)
      - record: job:container_memory:usage_bytes
        expr: sum(container_memory_usage_bytes) by (job)