Networking

Understanding Kubernetes networking concepts and implementation

Kubernetes networking addresses four primary concerns: container-to-container, pod-to-pod, pod-to-service, and external-to-service communication. The networking model in Kubernetes is designed to provide a flat network where all pods can communicate with each other without NAT, regardless of which node they are running on.

The Kubernetes networking model is built on these fundamental requirements:

All containers can communicate with all other containers without NAT
All nodes can communicate with all containers without NAT
The IP that a container sees itself as is the same IP that others see it as

This foundation creates a clean, backwards-compatible model where pods can be treated much like VMs or physical hosts from a port allocation, naming, service discovery, load balancing, application configuration, and migration perspective.

Pod Networking

Pod Network Model

Each pod gets unique IP address across the entire cluster
Containers within a pod share the pod's network namespace
Direct communication between pods across nodes without NAT
All containers in a pod share the same IP address and port space
Containers within a pod can communicate via localhost
Pods on the same node can communicate via pod IPs directly
Pods on different nodes communicate through the cluster network

Container Network Interface (CNI)

Network plugin architecture that defines how networking is implemented
Popular plugins: Calico, Flannel, Weave Net, Cilium, AWS VPC CNI
Implements pod networking according to the Kubernetes networking model
Handles IP allocation, routing, and network policy enforcement
Different CNI plugins offer various features:
- Calico: Network policy enforcement, high performance
- Flannel: Simple overlay network, easy to set up
- Cilium: eBPF-based networking with advanced security
- Weave Net: Mesh networking with encryption
- AWS VPC CNI: Native AWS VPC networking for EKS

Network Implementation

# Check which CNI plugin is deployed
kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'

# View CNI configuration
ls /etc/cni/net.d/

# Check pod networking
kubectl get pods -o wide

# Test connectivity between pods
kubectl exec -it pod-name -- ping other-pod-ip

Services

Services provide a stable endpoint to access a group of pods that provide the same functionality. They enable loose coupling between dependent pods by abstracting away the specific IP addresses of individual pods.

ClusterIP

The default service type that exposes the service on an internal IP within the cluster. This makes the service only reachable from within the cluster.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  labels:
    app: my-app
  annotations:
    service.kubernetes.io/topology-aware-hints: "auto"
spec:
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80           # Port exposed by the service
      targetPort: 9376   # Port on the pod to route traffic to
  type: ClusterIP
  sessionAffinity: None  # ClientIP for session stickiness
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  # Optional - fixed cluster IP
  # clusterIP: 10.96.10.10

Key aspects of ClusterIP:

Only accessible within the cluster
Provides load balancing across all matching pods
Stable internal IP that remains constant even as pods come and go
Used for internal communication between applications
Implemented by kube-proxy using iptables or IPVS rules
DNS name format: <service-name>.<namespace>.svc.cluster.local

NodePort

Exposes the service on each node's IP at a static port. Creates a ClusterIP service, which it routes to. Accessible from outside the cluster using <NodeIP>:<NodePort>.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
    - port: 80           # ClusterIP port
      targetPort: 9376   # Pod port
      nodePort: 30007    # Port on every node (30000-32767 range)
  externalTrafficPolicy: Cluster  # or Local for preserving client source IP

Key aspects of NodePort:

Makes service accessible from outside the cluster
Automatically creates a ClusterIP service
NodePort range is typically 30000-32767 (configurable)
Each node proxies the allocated port to the service
Useful for development or when external load balancers aren't available
Not recommended for production due to limited port range and security concerns

LoadBalancer

Exposes the service using a cloud provider's load balancer. Creates NodePort and ClusterIP services, which it routes to.

apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"  # Cloud-specific settings
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  externalTrafficPolicy: Cluster  # or Local for preserving client source IP
  loadBalancerSourceRanges:       # Restrict access to these IP ranges
    - 203.0.113.0/24
  # Optional fixed external IP
  # loadBalancerIP: 203.0.113.10  # Only on supported cloud providers

Key aspects of LoadBalancer:

Integrates with cloud provider's load balancing service
Automatically creates NodePort and ClusterIP services
Provides public IP address for external access
Distributes traffic among pods
Cloud provider provisions real load balancer (AWS ELB, GCP Load Balancer, etc.)
Most appropriate for production workloads needing external access
Can be costly as each service gets its own load balancer

ExternalName

Maps the service to the contents of the externalName field (e.g., foo.bar.example.com). Returns a CNAME record with the value.

apiVersion: v1
kind: Service
metadata:
  name: my-external-service
spec:
  type: ExternalName
  externalName: api.example.com

Key aspects of ExternalName:

No proxying involved, just DNS CNAME resolution
Used for connecting to external services
Provides an abstraction layer for external dependencies
Useful for migrating services to Kubernetes
No selectors, ports, or endpoints needed

Network Policies

Network Policies are application-centric constructs that specify how groups of pods are allowed to communicate with each other and with other network endpoints. They are essential for implementing security boundaries and enforcing the principle of least privilege in your cluster.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          role: frontend
    - namespaceSelector:
        matchLabels:
          project: myproject
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    ports:
    - protocol: TCP
      port: 6379
      endPort: 6400  # Optional range (Kubernetes 1.21+)
  egress:
  - to:
    - podSelector:
        matchLabels:
          role: monitoring
    ports:
    - protocol: TCP
      port: 5978

Key Components of Network Policies

podSelector: Determines which pods the policy applies to
policyTypes: Specifies whether policy applies to ingress, egress, or both
ingress/egress rules: Defines allowed traffic patterns
from/to selectors: Can select by pod, namespace, or IP block
ports: Specifies which ports traffic is allowed on

Default Deny All Traffic

# Deny all ingress traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

# Deny all egress traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress

Allow Traffic From Specific Namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-monitoring
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          purpose: monitoring

Note on Implementation

Network Policies require a network plugin that supports them, such as Calico, Cilium, or Weave Net. The default kubenet plugin does not support Network Policies.

# Check if your cluster supports Network Policies
kubectl run --rm -it --restart=Never busybox --image=busybox -- wget -O- http://nginx:80
# If policy is working, this should be blocked when appropriate policies are in place

Ingress Controllers

Ingress is an API object that manages external access to services in a cluster, typically HTTP/HTTPS. It provides load balancing, SSL termination, and name-based virtual hosting. Unlike Services, Ingress operates at the application layer (L7) of the network stack.

Ingress requires an Ingress controller to work. Multiple Ingress controllers are available, including NGINX, Traefik, HAProxy, and cloud provider-specific controllers.

Basic Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - http:
      paths:
      - path: /testpath
        pathType: Prefix     # Options: Exact, Prefix, or ImplementationSpecific
        backend:
          service:
            name: test
            port:
              number: 80

Ingress Controller Installation

Most clusters don't come with an Ingress controller installed by default. You need to deploy one:

# Install NGINX Ingress Controller with Helm
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install nginx-ingress ingress-nginx/ingress-nginx

# Verify installation
kubectl get pods -n ingress-nginx
kubectl get services -n ingress-nginx

TLS Configuration

SSL/TLS termination for secure connections:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
    - example.com
    - www.example.com
    secretName: example-tls-secret  # Secret containing tls.crt and tls.key
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

Creating a TLS secret:

# Generate self-signed certificate
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout tls.key -out tls.crt -subj "/CN=example.com"

# Create Kubernetes secret for TLS
kubectl create secret tls example-tls-secret \
  --key tls.key --cert tls.crt

Path-based Routing

Route traffic to different services based on URL path:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-based-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
      - path: /admin(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: admin-service
            port:
              number: 80
      - path: /(.*)
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80

Host-based Routing

Route traffic to different services based on hostname:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host-based-ingress
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 80
  - host: web.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80

Advanced Annotations

Different Ingress controllers support various annotations for advanced features:

metadata:
  annotations:
    # NGINX-specific annotations
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    
    # Rate limiting
    nginx.ingress.kubernetes.io/limit-rps: "10"
    
    # Authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    
    # Canary deployments
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"

DNS Service

Kubernetes provides a built-in DNS service for service discovery within the cluster. This DNS service allows pods to find and communicate with other services using consistent DNS names, regardless of the actual pod IP addresses.

CoreDNS

Default DNS service in Kubernetes since v1.13
Serves as the cluster's internal DNS server
Enables service discovery through DNS names
Provides Pod DNS resolution for containerized applications
Highly configurable and extensible with plugins
Implements service and pod DNS records according to Kubernetes specifications
Records automatically updated when services or pods change
Runs as Deployment in the kube-system namespace

# Check CoreDNS deployment
kubectl get deployment coredns -n kube-system

# View CoreDNS configuration
kubectl get configmap coredns -n kube-system -o yaml

# Debug DNS resolution from a pod
kubectl run -it --rm debug --image=alpine --restart=Never -- sh -c "apk add bind-tools && nslookup kubernetes.default.svc.cluster.local"

DNS Naming Convention

Services:
- Regular service: <service-name>.<namespace>.svc.cluster.local
- Headless service pods: <pod-name>.<service-name>.<namespace>.svc.cluster.local
- Example: nginx.default.svc.cluster.local
Pods:
- Full: <ip-with-dashes>.<namespace>.pod.cluster.local
- Example: 10-244-2-5.default.pod.cluster.local (for IP 10.244.2.5)

The .cluster.local suffix is the default domain but can be configured. Within the same namespace, services can be referenced by just <service-name>.

DNS Policies

DNS policies define how DNS resolution works for pods:

ClusterFirst (default):
- Queries that don't match the cluster domain suffix are forwarded to upstream nameservers
- Enables both cluster service lookup and external domain resolution
- Most common and recommended for typical workloads
spec: dnsPolicy: ClusterFirst
Default:
- Pod inherits the name resolution configuration from the node it runs on
- Uses the node's /etc/resolv.conf
- Doesn't use cluster DNS service
- Useful for pods that need to resolve external DNS exactly like the node
spec: dnsPolicy: Default
None:
- Ignores DNS settings from the Kubernetes environment
- Requires manually setting DNS via dnsConfig property
- Provides complete control over DNS configuration
spec: dnsPolicy: None dnsConfig: nameservers: - 8.8.8.8 - 1.1.1.1 searches: - example.com options: - name: ndots value: "5"
ClusterFirstWithHostNet:
- For pods running with hostNetwork: true
- Forces using cluster DNS instead of host DNS
- Needed when pods use host networking but still need to resolve cluster services
spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet

Custom DNS Configuration

You can customize DNS settings regardless of DNS policy:

apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-pod
spec:
  containers:
  - name: dns-example
    image: nginx
  dnsPolicy: ClusterFirst
  dnsConfig:
    nameservers:
      - 8.8.8.8
    searches:
      - ns1.svc.cluster.local
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "4"
      - name: timeout
        value: "3"

Headless Services for Direct Pod Access

When you need direct access to individual pods rather than load-balanced access:

apiVersion: v1
kind: Service
metadata:
  name: headless-service
spec:
  clusterIP: None  # This makes it headless
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80

With a headless service, DNS will return the pod IPs directly instead of the service IP.

Best Practices

Use appropriate service types
- Use ClusterIP for internal-only services
- Use NodePort only for development or edge cases
- Use LoadBalancer for production external services
- Use ExternalName for integrating with external services
- Consider Ingress for HTTP/HTTPS applications to consolidate multiple services
Implement network policies
- Start with a default deny policy and explicitly allow required traffic
- Create separate policies for ingress and egress
- Apply the principle of least privilege to all services
- Label pods and namespaces consistently for policy targeting
- Test network policies thoroughly before enforcing in production
- Example default deny policy:
  apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: - Ingress - Egress
Configure health checks for all services
- Implement readiness probes to control traffic routing
- Use liveness probes to restart unhealthy containers
- Configure appropriate probe timeouts and thresholds
- Make health check endpoints lightweight and reliable
- Consider startup probes for slow-starting applications
- Example:
  readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 15 periodSeconds: 10
Use meaningful DNS names and service discovery
- Create services with descriptive, consistent names
- Use headless services for stateful applications that need direct pod addressing
- Leverage DNS for service discovery instead of hardcoding IPs
- Use fully qualified domain names in critical paths
- Consider external DNS controllers for external-facing services
- Example:
  # Better name: 'auth-api' instead of 'service1' apiVersion: v1 kind: Service metadata: name: auth-api
Monitor network performance and connectivity
- Implement network monitoring with Prometheus metrics
- Track network latency between services
- Monitor service connection errors and timeouts
- Set up alerts for network issues
- Regularly test cross-node connectivity
- Use tools like ksniff or tcpdump for troubleshooting
- Example:
  # Install ksniff for packet capture kubectl sniff <pod-name> -n <namespace> -f "port 80"
Optimize for performance
- Use services with 'externalTrafficPolicy: Local' to preserve client source IPs and reduce hops
- Consider topology-aware service routing for multi-zone clusters
- Use the appropriate CNI plugin for your workload's performance needs
- Tune MTU settings for your network environment
- Example:
  apiVersion: v1 kind: Service metadata: name: web-service annotations: service.kubernetes.io/topology-aware-hints: "auto" spec: externalTrafficPolicy: Local
Secure external access properly
- Terminate TLS at the Ingress level
- Use cert-manager for automated certificate management
- Implement rate limiting for public endpoints
- Consider a service mesh for mTLS between services
- Use WAF (Web Application Firewall) for critical services
- Example:
  apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: example-cert spec: secretName: example-tls dnsNames: - example.com - www.example.com issuerRef: name: letsencrypt-prod kind: ClusterIssuer
Plan for DNS scalability
- Configure appropriate DNS caching
- Monitor CoreDNS performance and scale horizontally if needed
- Consider NodeLocal DNSCache for large clusters
- Set appropriate ndots value to reduce unnecessary lookups
- Example NodeLocal DNSCache:
  # Part of daemonset spec dnsConfig: options: - name: ndots value: "2"

Edit this page

Workloads

Understanding Kubernetes workload resources and their management

Storage

Understanding Kubernetes storage concepts, volumes, and persistence