Kubernetes Topology Aware Routing
Advanced guide to implementing zone-aware traffic management, latency-based optimization, and multi-region failover configurations for global service deployments
Introduction to Kubernetes Topology Aware Routing
Kubernetes Topology Aware Routing provides sophisticated traffic management capabilities based on the physical or logical topology of your cluster infrastructure.
Zone-aware Traffic Distribution
Route service traffic to pods in the same zone as the client
Latency Optimization
Minimize network latency by preferring local endpoints
Cross-zone Traffic Reduction
Lower inter-zone data transfer costs and bandwidth usage
Failure Isolation
Improve resilience by containing failures within topology boundaries
This comprehensive guide explores the architecture, implementation patterns, and operational best practices for topology-aware routing in Kubernetes, helping you optimize application performance and reduce costs in distributed environments.
Topology Aware Routing Architecture
Core Concepts
Kubernetes Topology Aware Routing builds on several foundational components:
┌─────────────────────────────────────────────────────────────────────┐
│ │
│ Kubernetes Cluster │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ │ │ │ │
│ │ Zone A │ │ Zone B │ │
│ │ │ │ │ │
│ │ ┌─────┐ ┌─────┐│ │ ┌─────┐ ┌─────┐│ │
│ │ │Pod A│ │Pod B││ │ │Pod C│ │Pod D││ │
│ │ └─────┘ └─────┘│ │ └─────┘ └─────┘│ │
│ │ │ │ │ │
│ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │
│ ┌────────▼───────────────────────────▼────────┐ │
│ │ │ │
│ │ Topology Aware Service │ │
│ │ │ │
│ └─────────────────────┬───────────────────────┘ │
│ │ │
│ │ │
│ ┌─────────────────────▼───────────────────────┐ │
│ │ │ │
│ │ EndpointSlice with Topology Hints │ │
│ │ │ │
│ └─────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
Topology Key Concepts
Topology Keys: Labels that identify infrastructure topology (zone, region, etc.)
EndpointSlice: Enhanced endpoint grouping with topology information
Service Topology: Service configuration that respects topology constraints
Topology Aware Hints: Feature gate that enables preferential local routing
Service Traffic Policy: Controls how traffic is distributed across endpoints
Enabling Topology Aware Routing
Prerequisites
Node Topology Labels
Verify your nodes have the required topology labels:
# Check node topology labels
kubectl get nodes --show-labels | grep topology.kubernetes.io
# Example output:
# worker-1 Ready <none> 90d v1.24.6 topology.kubernetes.io/zone=us-east-1a,topology.kubernetes.io/region=us-east-1
# worker-2 Ready <none> 90d v1.24.6 topology.kubernetes.io/zone=us-east-1b,topology.kubernetes.io/region=us-east-1
# worker-3 Ready <none> 90d v1.24.6 topology.kubernetes.io/zone=us-east-1c,topology.kubernetes.io/region=us-east-1
Add topology labels if they're missing:
# Add topology labels to nodes
kubectl label nodes worker-1 topology.kubernetes.io/zone=us-east-1a topology.kubernetes.io/region=us-east-1
kubectl label nodes worker-2 topology.kubernetes.io/zone=us-east-1b topology.kubernetes.io/region=us-east-1
kubectl label nodes worker-3 topology.kubernetes.io/zone=us-east-1c topology.kubernetes.io/region=us-east-1
Feature Gate Configuration
Enable the Topology Aware Hints feature gate in your cluster:
For kube-apiserver, kube-controller-manager, and kube-scheduler, add:
--feature-gates=TopologyAwareHints=true
For a kubeadm-managed cluster, modify the cluster configuration:
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
featureGates:
TopologyAwareHints: true
Implementing Zone-Aware Routing
Basic Topology Aware Service
Create a service with topology awareness enabled:
apiVersion: v1
kind: Service
metadata:
name: topology-aware-service
spec:
selector:
app: myapp
ports:
- port: 80
targetPort: 8080
topologyKeys:
- "topology.kubernetes.io/zone"
- "topology.kubernetes.io/region"
- "*"
This configuration will:
- First attempt to route traffic to endpoints in the same zone
- If no endpoints exist in the same zone, try endpoints in the same region
- If no regional endpoints are available, fall back to any endpoint
For newer Kubernetes versions, use the topology-aware hints approach:
apiVersion: v1
kind: Service
metadata:
name: topology-aware-service
annotations:
service.kubernetes.io/topology-aware-hints: "auto"
spec:
selector:
app: myapp
ports:
- port: 80
targetPort: 8080
type: ClusterIP
The "auto" setting enables Kubernetes to automatically distribute traffic based on endpoint topology.
Inspect how EndpointSlices incorporate topology information:
kubectl get endpointslices -o yaml
Example EndpointSlice with topology hints:
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
name: topology-aware-service-abc123
labels:
kubernetes.io/service-name: topology-aware-service
addressType: IPv4
ports:
- name: http
port: 8080
protocol: TCP
endpoints:
- addresses:
- "10.0.1.1"
conditions:
ready: true
hostname: pod-1
nodeName: worker-1
zone: "us-east-1a"
hints:
forZones:
- name: "us-east-1a"
- addresses:
- "10.0.2.2"
conditions:
ready: true
hostname: pod-2
nodeName: worker-2
zone: "us-east-1b"
hints:
forZones:
- name: "us-east-1b"
Implement sophisticated load balancing across zones while respecting topology:
apiVersion: v1
kind: Service
metadata:
name: multi-zone-service
annotations:
service.kubernetes.io/topology-aware-hints: "auto"
service.kubernetes.io/topology-mode: "Auto"
spec:
selector:
app: multi-zone-app
ports:
- port: 80
targetPort: 8080
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
Configure cross-zone failover for high availability:
apiVersion: v1
kind: Service
metadata:
name: failover-service
annotations:
service.kubernetes.io/topology-aware-hints: "auto"
service.kubernetes.io/topology-mode: "Auto"
service.kubernetes.io/local-zone-recovery: "true"
spec:
selector:
app: resilient-app
ports:
- port: 80
targetPort: 8080
Implement weighted traffic distribution across zones:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: weighted-ingress
annotations:
nginx.ingress.kubernetes.io/service-weight: |
service-zone-a=70,service-zone-b=30
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-zone-a
port:
number: 80
Implement global load balancing across multiple regions:
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ Global Load Balancer (DNS/GSLB) │
│ │
└───────────────┬─────────────────────────────────┬─────────────────────┬─┘
│ │ │
▼ ▼ ▼
┌───────────────────────────┐ ┌───────────────────────────┐ ┌───────────────────────┐
│ │ │ │ │ │
│ Region: us-east-1 │ │ Region: eu-west-1 │ │ Region: ap-south-1 │
│ │ │ │ │ │
│ ┌─────────┐ ┌─────────┐ │ │ ┌─────────┐ ┌─────────┐ │ │ ┌─────────┐ │
│ │ Zone A │ │ Zone B │ │ │ │ Zone A │ │ Zone B │ │ │ │ Zone A │ │
│ └────┬────┘ └────┬────┘ │ │ └────┬────┘ └────┬────┘ │ │ └────┬────┘ │
│ │ │ │ │ │ │ │ │ │ │
│ ┌────▼───────────▼────┐ │ │ ┌────▼───────────▼────┐ │ │ ┌────▼────┐ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ Regional Ingress │ │ │ │ Regional Ingress │ │ │ │ Regional │ │
│ │ │ │ │ │ │ │ │ │ Ingress │ │
│ └─────────────────────┘ │ │ └─────────────────────┘ │ │ └─────────┘ │
│ │ │ │ │ │
└───────────────────────────┘ └───────────────────────────┘ └───────────────────────┘
Implement federation for multi-cluster topology routing:
apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
name: global-service
namespace: default
spec:
template:
spec:
selector:
app: global-app
ports:
- port: 80
targetPort: 8080
placement:
clusters:
- name: cluster-us-east
- name: cluster-eu-west
- name: cluster-ap-south
overrides:
- clusterName: cluster-us-east
clusterOverrides:
- path: "metadata.annotations"
value:
service.kubernetes.io/topology-aware-hints: "auto"
- clusterName: cluster-eu-west
clusterOverrides:
- path: "metadata.annotations"
value:
service.kubernetes.io/topology-aware-hints: "auto"
Implement regional traffic affinity with global failover capabilities:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: global-destination-rule
spec:
host: global-service
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
failover:
- from: us-east-1
to: [us-west-1, eu-west-1]
- from: eu-west-1
to: [eu-central-1, us-east-1]
failoverPriority:
- "topology.kubernetes.io/region"
- "topology.kubernetes.io/zone"
outlierDetection:
consecutive5xxErrors: 3
interval: 30s
baseEjectionTime: 30s
Implement advanced latency-based routing with service mesh tools:
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: latency-optimized-service
spec:
host: my-service
trafficPolicy:
loadBalancer:
localityLbSetting:
enabled: true
distribute:
- from: us-east/us-east-1/*
to:
"us-east/us-east-1/*": 80
"us-east/us-east-2/*": 20
failover:
- from: us-east
to: [us-west, eu-west]
Configure routing to minimize cross-zone data transfer costs:
apiVersion: v1
kind: Service
metadata:
name: cost-optimized-service
annotations:
service.kubernetes.io/topology-aware-hints: "auto"
service.kubernetes.io/topology-mode: "Auto"
service.kubernetes.io/prefer-local-traffic: "true"
spec:
selector:
app: data-intensive-app
ports:
- port: 80
targetPort: 8080
Monitor the effectiveness of topology-aware routing with these metrics:
# Using Prometheus queries for topology metrics
kubectl -n monitoring exec -it prometheus-k8s-0 -- curl -s 'localhost:9090/api/v1/query?query=service_traffic_split{service="topology-aware-service"}' | jq
Key metrics to track:
- Cross-zone traffic percentage: Measures traffic routed outside local zones
- Zone latency: Measures response time differences between zones
- Endpoint distribution: Tracks how traffic is distributed across endpoints
- Zone failover events: Records when traffic shifts between zones
If traffic isn't following topology hints:
- Verify feature gate is enabled:
kubectl get configmap -n kube-system kube-proxy -o yaml | grep "TopologyAwareHints"
- Check EndpointSlice configuration:
kubectl get endpointslices -o custom-columns=NAME:.metadata.name,HINTS:.endpoints[*].hints
- Validate node labels:
kubectl get nodes --show-labels | grep topology
If traffic is heavily skewed to certain zones:
- Check pod distribution:
kubectl get pods -o wide | grep app=myapp
- Adjust deployment to ensure even zone distribution:
apiVersion: apps/v1
kind: Deployment
metadata:
name: zone-balanced-app
spec:
replicas: 6
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: myapp
Implement topology-aware routing for a global e-commerce application:
# Deployment ensuring zone distribution
apiVersion: apps/v1
kind: Deployment
metadata:
name: ecommerce-api
spec:
replicas: 9
selector:
matchLabels:
app: ecommerce-api
template:
metadata:
labels:
app: ecommerce-api
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: ecommerce-api
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- ecommerce-api
topologyKey: topology.kubernetes.io/zone
containers:
- name: api-server
image: ecommerce/api-server:v1.0
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi
# Service with topology-aware routing
apiVersion: v1
kind: Service
metadata:
name: ecommerce-api
annotations:
service.kubernetes.io/topology-aware-hints: "auto"
spec:
selector:
app: ecommerce-api
ports:
- port: 80
targetPort: 8080
sessionAffinity: ClientIP
Configure a financial services application with strict topology requirements:
# Regional Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: financial-app-us-east
namespace: financial-services
spec:
replicas: 6
selector:
matchLabels:
app: financial-app
region: us-east
template:
metadata:
labels:
app: financial-app
region: us-east
spec:
nodeSelector:
topology.kubernetes.io/region: us-east-1
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: financial-app
region: us-east
containers:
- name: financial-service
image: financial/service:v2.3
env:
- name: REGION
value: "us-east-1"
# Global Service with regional failover
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: financial-global-service
namespace: financial-services
spec:
hosts:
- "api.financial-app.com"
gateways:
- financial-gateway
http:
- route:
- destination:
host: financial-app-us-east
port:
number: 80
weight: 100
- destination:
host: financial-app-eu-west
port:
number: 80
weight: 0
retries:
attempts: 3
perTryTimeout: 500ms
fault:
abort:
percentage:
value: 0
httpStatus: 503
1
Consistent labeling: Ensure all nodes have accurate topology labels
2
Balanced deployments: Use topology spread constraints for even distribution
3
Graceful degradation: Configure sensible failover paths
4
Cost consciousness: Prioritize local routing to minimize cross-zone charges
5
Performance monitoring: Track and optimize topology-related metrics
Properly labeled all nodes with topology information
Enabled required feature gates in all cluster components
Configured services with appropriate topology annotations
Set up monitoring for cross-zone traffic and latency
Implemented topology spread constraints for deployments
Tested failover scenarios between zones and regions
Documented topology architecture and failover paths
Kubernetes Topology Aware Routing provides powerful capabilities for optimizing application performance, reducing costs, and enhancing resilience in distributed environments. By intelligently routing traffic based on infrastructure topology, organizations can deliver lower-latency experiences to users while minimizing cross-zone data transfer expenses.
Improved Performance
Lower latency through local endpoint preference
Cost Optimization
Reduced cross-zone data transfer charges
Enhanced Resilience
Better fault isolation during partial outages
Global Scalability
Simplified management of multi-region deployments
::div{class="bg-blue-100 dark:bg-blue-800 p-2 rounded-
On this page
- Introduction to Kubernetes Topology Aware Routing
- Topology Aware Routing Architecture
- Enabling Topology Aware Routing
- Implementing Zone-Aware Routing
- Advanced Routing Patterns
- Multi-Region Architecture
- Performance Optimization
- Monitoring and Troubleshooting
- Practical Implementation Examples
- Best Practices and Recommendations
- Conclusion