Advanced Kubernetes
through the F1 lens.

Stage 01 probes + requests/limits

Application health and resource management

The problem

The concept

Probes. You have already seen all three in the basics guide; here the focus is on behavior under failure.

• Liveness asks "has the container frozen?". If it fails the number of times defined in failureThreshold, the kubelet restarts the container. It is meant for unrecoverable freezes (deadlock), not for passing slowness, otherwise the app enters a cycle of restarts. If the process simply dies, what recreates the container is the Pod's restartPolicy (the default is Always), with no involvement of the liveness; the liveness covers the case where the process is still alive but frozen.
• Readiness asks "can it receive requests?". If it fails, the Pod is removed from the Service endpoints and is not restarted. It comes back on its own when it passes again. It is what guarantees a rollout without errors for the user.
• Startup asks "has it finished starting?". Until it passes, liveness and readiness are suspended. It protects slow-booting apps that would otherwise be killed before they became ready.

Requests and limits. Every container can declare two values per resource:

• requests is what the container *reserves*. The scheduler uses this value to decide on which node the Pod fits. It is the guaranteed floor.
• limits is the ceiling it cannot exceed. Going over the memory limit makes the kernel shut down the container (OOMKill). Going over the CPU limit kills nothing: the kernel slows the process down (throttling) to keep it within the quota.

CPU is measured in millicores (100m is 0.1 CPU) and memory in bytes with binary suffixes (128Mi).

QoS class. From the combination of requests and limits comes the Pod's QoS class, which weighs on the decision of who leaves first under pressure:

• Guaranteed: all containers have requests equal to limits, for both CPU and memory.
• Burstable: at least one container declares a request or a limit, but the Pod does not qualify as Guaranteed. It uses idle slack and yields under contention.
• BestEffort: no container declares requests or limits. It is the first to be sacrificed.

A counterintuitive point: a CPU limit can worsen latency. The Linux kernel applies the limit through quotas in periods of 100 ms (CFS quota). A service with several threads can spend the whole quota of the period in a short burst and stand still until the next period, even with low average usage. Buffer publicly reported that it removed the CPU limits of latency-sensitive services and saw the p99 (the latency below which 99% of requests fall) drop significantly (the source report mentions 5 to 10 times). The practical lesson: in a cluster shared by several teams, limits provide isolation; in a latency-sensitive service, it is worth considering keeping the request and removing the CPU limit, keeping the memory limit.

Availability and signals. The availability of a service is A = MTBF / (MTBF + MTTR): the mean time between failures divided by that same time plus the mean time to recovery. With independent replicas, failure probabilities multiply: if each Pod is down 1% of the time, the chance of two Pods being down at the same time is 0.01 × 0.01 = 0.0001, that is, 0.01%, and the set reaches about 99.99% availability, as long as the failures are independent. The Service works as a "logical OR", because a single Ready Pod is enough. To know whether the service is truly healthy, Google SRE proposes the four *golden signals*: latency, traffic, errors and saturation. Probes cover errors and part of saturation; for the rest, Prometheus and Grafana are used.

Key concepts

Worked examples

 # deploy-saude.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3                          # three copies, as in the basics guide
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api                       # must match the selector
    spec:
      containers:
        - name: api
          image: stefanprodan/podinfo:6.14.1   # fixed version; confirm the tag with the command above
          ports:
            - containerPort: 9898
          startupProbe:                # protects a slow boot
            httpGet:
              path: /healthz
              port: 9898
            failureThreshold: 30       # up to 30 attempts
            periodSeconds: 10          # every 10s: up to 5min to start
          livenessProbe:               # frozen? restart the container
            httpGet:
              path: /healthz
              port: 9898
            periodSeconds: 10
            failureThreshold: 3        # 3 failures in a row before restarting
          readinessProbe:              # ready? joins the Service
            httpGet:
              path: /readyz
              port: 9898
            periodSeconds: 5
          resources:
            requests:                  # reservation used by the scheduler
              cpu: 100m
              memory: 64Mi
            limits:
              memory: 128Mi            # memory ceiling; no CPU limit, to avoid throttling
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api                           # finds the Pods by the label
  ports:
    - port: 80
      targetPort: 9898

Essential commands for this stage

Hands-on

1. Apply deploy-saude.yaml and confirm 3 Running Pods with kubectl get pods -l app=api. Run the qosClass command and observe Burstable.
2. Note down the names of two Pods from the list (<pod-1> and <pod-2>). Turn off the readiness of the first with kubectl exec <pod-1> -- curl -s -X POST localhost:9898/readyz/disable. Observe READY 0/1 with RESTARTS 0 and one fewer endpoint in kubectl get endpoints api.
3. Bring down the process of the second Pod with kubectl exec <pod-2> -- curl -s localhost:9898/panic and follow with kubectl get pods -w. Observe RESTARTS go up for <pod-2>. What restarted it was the restartPolicy: the process died and the kubelet recreated the container, with no involvement of the liveness. If the first /panic does not bring the process down, repeat it.
4. Apply liveness-falha.yaml and follow with kubectl get pod liveness-falha -w. Observe RESTARTS go up after 30 to 40 seconds, although the process remains alive, and confirm in kubectl describe pod liveness-falha the pair Liveness probe failed and Killing. This is the liveness acting on an app that is alive and frozen.
5. Apply pods-qos.yaml and confirm the three classes with the custom-columns. Observe that the class is calculated by Kubernetes from the requests and limits, without you declaring it.
6. Apply pod-oom.yaml and observe the OOMKilled status (confirm with describe pod and look for Reason: OOMKilled).
7. Clean up: kubectl delete -f deploy-saude.yaml -f pods-qos.yaml -f pod-oom.yaml -f liveness-falha.yaml.

Stage 02 taints + affinity + spread

Advanced scheduling and node behavior

The problem

The concept

Taints and tolerations: repelling. The *taint* is a mark placed on the node that says "do not schedule Pods here, unless they tolerate this mark". The *toleration* is the declaration on the Pod: "I accept that mark, I can go". A taint has a key, a value and an effect:

• NoSchedule: does not schedule Pods that do not tolerate it; those already running there stay.
• PreferNoSchedule: avoids, but is not absolute.
• NoExecute: besides blocking new ones, it evicts the Pods that do not tolerate it and were already on the node (useful for emptying a node).

A crucial point: the taint only repels, it does not attract. A Pod that tolerates the GPU node's taint can go to the GPU node or to an ordinary node. To guarantee that it goes *only* to the GPU, combine it with node affinity.

Node affinity: attracting. It is the positive counterpart: "I want to go to nodes with this label". There are two modes, written in the fields requiredDuringSchedulingIgnoredDuringExecution (mandatory: without the label, the Pod stays Pending) and preferredDuringSchedulingIgnoredDuringExecution (preference, with a weight: if there is none, it is scheduled elsewhere). The IgnoredDuringExecution part indicates that the rule applies only at scheduling time; if the node's label changes later, the Pod is not moved.

Pod affinity and anti-affinity: relating Pods to each other. Instead of looking at node labels, they look at the Pods that are already there. *Affinity* brings together (cache close to the frontend, in the same zone); *anti-affinity* separates (do not put two database replicas on the same node). The zone is a separate physical area inside the same region of the cloud provider, with its own power and network. The Kubernetes documentation does not recommend using inter-pod rules in clusters of several hundred nodes, because the calculation is expensive for the scheduler.

Topology Spread Constraints: spreading with a numeric guarantee. It is the preferred alternative for distributing replicas. You declare a topologyKey (the grouping criterion, for example kubernetes.io/hostname for nodes or topology.kubernetes.io/zone for zones), a maxSkew (the maximum allowed difference in the number of Pods between two groups) and whenUnsatisfiable: DoNotSchedule (strict, the Pod stays Pending if it would violate the rule) or ScheduleAnyway (preference). With 4 replicas, 2 nodes and maxSkew: 1, the only accepted distribution is 2 and 2; 3 and 1 is rejected, because the difference would be 2.

Eviction under pressure: the node's self-protection. When the node runs out of memory, disk or PIDs, the kubelet acts before the collapse. First it tries to free space (dead containers, unused images). If that is not enough, it starts evicting Pods. The order follows the documentation: first the Pods whose usage of the scarce resource exceeds the request, then the priority (PriorityClass), then the size of the usage relative to the request. In practice, BestEffort and Burstable Pods that went over their requests leave first, and Guaranteed and high-priority Pods leave last. Details that matter:

• Eviction under resource pressure ignores the PodDisruptionBudget, because it is an emergency.
• The kubelet parameter eviction-minimum-reclaim makes the kubelet free a margin beyond the threshold, to avoid a domino effect of repeated evictions.
• Eviction is not OOMKill. OOMKill is the kernel shutting down *a container* that went over its memory limit; eviction is the kubelet removing *Pods* to save the *node*.

Planned maintenance: cordon, drain and PodDisruptionBudget. When you need to empty a node on purpose (upgrade, machine replacement), you use kubectl cordon (marks the node as unschedulable) and kubectl drain (cordon, plus orderly eviction of the Pods). This kind of interruption, decided and scheduled by someone, is called a voluntary disruption; an interruption that happens unasked (a node out of memory, a machine taken back by the provider) is involuntary. The PodDisruptionBudget (PDB) defines how many Pods of an application may be unavailable during a voluntary disruption, in minAvailable (the minimum that must stay up) or maxUnavailable (the maximum that may leave). The drain respects the PDB: if removing a Pod would violate the budget, the eviction is refused and retried until it is possible. Eviction under resource pressure, described above, is involuntary and does not respect the PDB.

PriorityClass and the nature of the scheduler. The PriorityClass assigns a numeric priority to Pods. System Pods (system-cluster-critical and system-node-critical) have very high priority and are almost never evicted. Allocating Pods onto nodes with multiple constraints is a bin packing problem, NP-hard; the scheduler uses a greedy two-phase heuristic (it filters the nodes that fit, scores the ones left, picks the best), fast but not necessarily optimal.

Key concepts

Worked examples

 # pod-gpu.yaml
 # Preliminary step, done once by whoever administers the cluster:
 #   kubectl taint nodes gpu-node-1 nvidia.com/gpu=true:NoSchedule
 #   kubectl label nodes gpu-node-1 accelerator=nvidia-gpu
apiVersion: v1
kind: Pod
metadata:
  name: treino-ml
spec:
  tolerations:                       # accepts the GPU node's taint
    - key: "nvidia.com/gpu"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  affinity:
    nodeAffinity:                    # AND requires going to a node with a GPU
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: "accelerator"
                operator: In
                values: ["nvidia-gpu"]
  containers:
    - name: treino
      image: myregistry/treino:1.0   # fictitious image
      resources:
        limits:
          nvidia.com/gpu: 1          # asks the device plugin for one GPU

Essential commands for this stage

Hands-on

1. If the default Minikube profile is running, run minikube stop. Bring up the 2-node cluster (minikube start -p lab --nodes 2), apply the taint and the label on lab-m02 and apply the two Pods from the example. Observe sem-toleration in Pending and the message untolerated taint in the describe; observe com-toleration in Running on lab-m02.
2. Remove the taint (kubectl taint nodes lab-m02 dedicated=ml:NoSchedule-), delete the two Pods and apply web-spread.yaml and pdb-web.yaml. Observe with -o wide the 4 Pods split 2 and 2 and, in kubectl get pdb web, the value 1 in ALLOWED DISRUPTIONS.
3. Delete web and pdb-web with kubectl delete -f web-spread.yaml -f pdb-web.yaml, apply pdb-drain.yaml, run the drain of lab-m02 and observe the refusal Cannot evict pod as it would violate the pod's disruption budget. Interrupt with Ctrl+C and run kubectl uncordon lab-m02.
4. Run kubectl describe node lab and observe the Conditions section: MemoryPressure, DiskPressure and PIDPressure at False are the healthy state. Eviction under pressure (which ignores the PDB) only happens when one of them becomes True; it is not reproduced here because it would require exhausting the node.
5. Clean up: minikube delete -p lab. This removal also deletes the lab context from kubectl, so return to the default cluster with minikube start (if you stopped the default profile in step 1) or with kubectl config use-context minikube, and confirm with kubectl get nodes before moving on to Stage 03.

Stage 03 update strategies

Update strategies and rollouts

The problem

The concept

RollingUpdate (the default) replaces Pods gradually, keeping the service up. The Deployment creates a new ReplicaSet with the new version and shrinks the old one as the new Pods become ready. Two parameters control the pace of the swap:

• maxUnavailable: the capacity floor, that is, how many Pods may be unavailable during the swap. Default: 25%.
• maxSurge: the extra slack, that is, how many Pods *more* than desired may exist temporarily. Default: 25%.

With N replicas and the default values, during the rollout there are at least about 75% of Pods available and at most about 125% of Pods running. Kubernetes rounds maxSurge up and maxUnavailable down; with 10 replicas, the default results in 3 surge and 2 unavailable, that is, at least 8 available and at most 13 running. The two parameters cannot both be zero, because the swap would have no way to progress.

Recreate shuts down *all* the Pods of the old version before bringing up those of the new one. There is downtime, but the two versions never coexist. It is the choice when v1 and v2 are incompatible.

Tuning is a trade-off. A high maxUnavailable allows more Pods to be swapped at the same time, which speeds up the swap, but reduces the capacity serving users during it, and the request queue grows. In queueing theory, λ is the arrival rate of requests, μ is how much each Pod serves per second and c is the number of Pods; the utilization ρ = λ/(cμ) is the fraction of capacity in use. With fewer Pods (smaller c), ρ rises, and latency grows very fast when ρ approaches 1. A high maxSurge also speeds things up, but doubles the number of Pods and connections, and that can saturate the database. Two useful recipes:

• Availability first: maxUnavailable: 0 (never reduces capacity) and maxSurge within the cluster's resource budget.
• Shortest time under a budget: choose maxSurge and maxUnavailable as high as possible, so that the peak of Pods ((1 + surge) × N) fits in the resource budget and the floor ((1 - unavailable) × N) does not degrade the service beyond what is acceptable.

Safeguard tools:

• minReadySeconds (default 0): how many seconds a Pod must remain Ready before it counts as available. It prevents moving ahead with a Pod that fails right after starting.
• progressDeadlineSeconds (default 600): if the rollout does not progress in that time, the Deployment receives the condition ProgressDeadlineExceeded. The Deployment does not roll back on its own; you do it (or a tool does).
• revisionHistoryLimit (default 10): how many old revisions (ReplicaSets) to keep for rollback.

The rollout is only triggered by changes in .spec.template. Changing the number of replicas or an annotation of the Deployment itself does not count.

More sophisticated strategies. Blue/Green and canary (next stages) are not native to the Deployment object. Controllers such as Argo Rollouts and Flagger implement them, with promotion steps and automatic analysis of metrics.

Key concepts

RollingUpdate with availability first

 # myapp.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  annotations:
    kubernetes.io/change-cause: "nginx 1.26"   # shows up in the rollout history
spec:
  replicas: 4
  minReadySeconds: 10                  # the Pod must stay Ready for 10s to count as available
  progressDeadlineSeconds: 600         # default: 10 minutes for the rollout to progress
  revisionHistoryLimit: 10             # default: keeps 10 revisions for rollback
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0                # never reduces capacity
      maxSurge: 1                      # creates 1 extra Pod at a time
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: app
          image: nginx:1.26
          readinessProbe:              # without readiness, the Pod counts as ready as soon as it starts
            httpGet:
              path: /
              port: 80

Essential commands for this stage

Hands-on

1. Apply myapp.yaml and confirm 4 Running Pods. In a second terminal, run kubectl get pods -w.
2. Run kubectl set image deployment/myapp app=nginx:1.27. Observe in the second terminal that there are always 4 ready Pods and that at most 1 extra Pod appears: it is maxUnavailable: 0 with maxSurge: 1 in action.
3. Record the change with kubectl annotate and confirm in kubectl rollout history deployment/myapp that revision 2 shows nginx 1.27.
4. Shorten the deadline with the patch of progressDeadlineSeconds: 30, swap the image to nginx:nao-existe and run kubectl rollout status deployment/myapp. Observe the message exceeded its progress deadline and confirm the exit code with echo $?.
5. Observe with kubectl get pods that the old Pods are still Running: the service never went down. Go back with kubectl rollout undo deployment/myapp.
6. Clean up: kubectl delete -f myapp.yaml.

Stage 04 packages + values

Package manager with Helm Charts

The problem

The concept

Helm works as the package manager of Kubernetes (comparable to apt or yum). A chart has a simple structure:

• `Chart.yaml`: package metadata. version is the version of the *chart*; appVersion is the version of the packaged *application*. Both follow SemVer, and separating them allows the chart to evolve without changing the app and vice versa.
• `values.yaml`: the configurable default values (replicas, image, port, resources). Everything that varies by environment lives here; each environment uses its own values file, with -f.
• `templates/`: the manifests written as Go language templates, with Sprig functions. {{ .Values.replicaCount }} inserts a value; files such as _helpers.tpl gather reusable snippets (names and labels).

A Helm template behaves like a function: given the same values, it produces the same YAML, and you check the result before applying with helm template. This gives predictability and makes testing possible. Functions such as lookup, now and randAlphaNum break this rule (they read the cluster or generate different values on each run), so use them consciously.

The life cycle of a release. Each installation of a chart is a release, with a name and a history:

• helm install creates the release and records the state as a Secret in the namespace (sh.helm.release.v1.<name>.v<revision>).
• helm upgrade renders the templates again, calculates the differences from what is installed, applies only what changed and creates a new revision. Running helm upgrade without changing anything does not alter the cluster's resources, but Helm records a new revision anyway.
• helm rollback <release> <revision> goes back to the state of an earlier revision, and the rollback also generates a new revision. The old revisions are not altered, and that is what makes the rollback reliable.

Helm 3 has no central server (the old Tiller was removed). The client uses the credentials of your kubeconfig, so the user's RBAC limits what Helm can do.

Governance at scale. Charts follow SemVer (a major version change indicates a break and calls for human review), are distributed through repositories (Artifact Hub, private repositories) and can be signed (GPG, with a .prov file), which protects the supply chain. A chart should embed least-privilege RBAC (dedicated ServiceAccounts, limited Roles). Subcharts allow composition (an "umbrella" chart that joins the app and the database), and values.schema.json validates the values like a type schema. In GitOps flows (Argo CD, Flux), the chart lives in Git, synchronization is automatic and each release becomes an auditable commit.

Key concepts

A minimal chart, from scratch

 # Chart.yaml
apiVersion: v2              # version of the Helm 3 chart format
name: meuapp
description: Chart de estudo
type: application
version: 0.1.0              # chart version
appVersion: "1.26"          # version of the packaged application (informational)

Essential commands for this stage

Hands-on

1. Create the folder meuapp/ with the three files, run helm lint ./meuapp and helm template web ./meuapp --set replicaCount=3. Observe replicas: 3 in the rendered YAML: the values were overridden without touching the file.
2. Run helm install web ./meuapp and kubectl get secret -l owner=helm. Observe the Secret that holds revision 1.
3. Run helm upgrade web ./meuapp --set replicaCount=3, then helm rollback web 1 and helm history web. Observe that the rollback created a new revision (number 3) instead of erasing number 2.
4. Run the upgrade with --set image.tag=nao-existe --atomic --timeout 30s. Observe the error UPGRADE FAILED ... rolled back and that kubectl get deploy web still shows the old Pods healthy.
5. Clean up: helm uninstall web.

Stage 05 whole-environment switch

Blue/Green deployment

The problem

The concept

You keep two identical production environments. Blue is the current version, serving 100% of the traffic. Green is the new version, with all Pods ready, but without traffic. When Green is validated, you switch the traffic to it. If there is a problem, you switch back.

In Kubernetes, the two sets of Pods are two Deployments, with different labels (env: blue and env: green, besides the common app). A Service points to one of them through the selector. The promotion is a patch on the Service's selector: the endpoints controller recalculates the Service's list of Pods and kube-proxy reconfigures the routing. The rollback is the inverse patch. Since the old Pods are still up (just without traffic), going back requires no redeployment. A practical note: the switch applies to new connections; connections that are already open (for example, HTTP keep-alive) may stay on the old Pods until they are closed.

Formally, it is a transition from one state to another with a single operation, and the rollback is the same operation in the opposite direction. If v2 fails, the damage is limited to the time until detection: reverting at once reduces that time to the minimum.

Pitfalls:

• Data and schema compatibility. It is the biggest practical challenge. If Green alters the database in an incompatible way, going back to Blue breaks. The solution is to make the change in two phases (a pattern known as expand/contract): first migrate the database to support *both* versions, validate, and only then switch the code. Without this, data written by Green can render Blue unusable.
• Cost. Two complete environments mean about 100% of extra capacity during the transition. Good practice: reduce the idle environment to zero replicas after the new version stabilizes.
• Connection storm and cold cache. Switching 100% of the traffic at once suddenly throws the whole load onto Green, and cold caches can generate a spike. One mitigation is to warm Green up with shadow traffic beforehand.

Classic antipattern: failing to treat Green as production all the time. If you accumulate changes waiting for the "right moment" to switch, the jump between versions becomes large and risky. True Blue/Green means deploying continuously to the idle environment and flipping the switch when there is confidence.

Tools. Argo Rollouts supports blueGreen natively (with a preview service and promotion that is manual or automatic by metrics); Flagger integrates with a service mesh. The pattern is also called Red/Black in contexts such as Netflix and Spinnaker.

Key concepts

A Service that points to blue and then to green

 # bluegreen.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      env: blue
  template:
    metadata:
      labels:
        app: myapp
        env: blue                     # identifies the environment
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-text=blue v1"]     # answers with this text, on port 5678
          ports:
            - containerPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
      env: green
  template:
    metadata:
      labels:
        app: myapp
        env: green
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-text=green v2"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
    env: blue                         # the key: switch to green on promotion
  ports:
    - port: 80
      targetPort: 5678

Essential commands for this stage

Hands-on

1. Apply bluegreen.yaml, run kubectl get endpoints myapp-service and the wget through the cliente Pod. Observe blue v1 and only the 2 IPs of the blue Pods.
2. Do the promotion with the kubectl patch from the example and query again. Observe green v2 and the 2 IPs of the green Pods in get endpoints, without any Pod having been created or removed.
3. Do the rollback with the inverse patch and observe blue v1 back, because the blue Pods kept existing.
4. Scale myapp-blue to 0, run kubectl get endpoints myapp-service and observe that, with the selector on blue, the Service was left without endpoints (<none>). This is why the shutdown of the old environment should only be done after promoting and stabilizing the new one.
5. Clean up: kubectl delete -f bluegreen.yaml.

Stage 06 progressive traffic

Canary deployment

The problem

The concept

In a canary, you keep both versions running and shift the traffic gradually (for example, 1%, 5%, 25%, 50% and 100%), validating real metrics at each step. If something degrades, the impact was restricted to the few percent and you revert. In contrast, Blue/Green is "all or nothing"; the canary is progressive.

Two ways to split the traffic:

1. By replica ratio, natively. Two Deployments with a label in common (for example app: myapp), and a Service that selects only by that label. The Service distributes requests among all the Pods, so 4 stable Pods and 1 canary result in about 20% of the traffic on the canary. It is simple, requires nothing beyond Kubernetes, but the granularity depends on the number of replicas (for 1% you would need 100 Pods).
2. By weight, with a service mesh or an ingress that supports weights. With Istio or Linkerd, or with a compatible ingress controller, you define exact percentages (for example 90% and 10%), independent of the number of replicas. In Istio, a VirtualService splits the traffic between *subsets* (v1 and v2), which are declared in a DestinationRule from the Pods' labels.

Metric-driven decision. What decides whether to promote or revert are real-time metrics: error rate (5xx), latency (p95 and p99, the latencies below which 95% and 99% of requests fall), resource usage, business metrics. This closes a feedback loop that can be automated: if the canary's error stays below the threshold, the weight increases; if it violates it, it goes back to v1.

Tools. Argo Rollouts and Flagger implement the steps (weights, pauses, analyses) as Kubernetes objects. In Argo Rollouts, the Rollout object replaces the Deployment and declares the strategy in steps. Without a configured traffic router (trafficRouting), the setWeight is approximated through the number of replicas, the same as the native technique; with Istio or a compatible ingress, the weight is exact.

Theory vocabulary, to know it exists:

• Continuous A/B testing: the canary is a statistical experiment between v1 (control) and v2. The null hypothesis is "there is no difference"; it is evaluated with significance tests (z-test of proportions, chi-squared, Mann-Whitney). Kayenta, used with Spinnaker at Netflix, performs canary analysis with the Mann-Whitney test.
• Sequential testing (Wald's SPRT): evaluates continuously and stops early when there is enough evidence to promote or abort, reducing user exposure.
• Queueing theory: even 10% of the traffic can degrade latency if those 10% nearly saturate the few Pods of v2. The waiting time grows very fast near saturation, which also serves to estimate whether v2 can handle 100%.
• Multi-armed bandits (Thompson Sampling): allocate traffic adaptively, balancing testing and favoring the best version.

Reported cases. The source material mentions Shopify (about 5% of the traffic for a few minutes, with automatic analysis before going to 100%), Netflix (automated canary analysis with Kayenta in Spinnaker) and Nubank (feature flags combined with canary). The exact numbers come from the original report and were not verified here. The trend is to guide the canary by SLOs (brake if the error target is violated).

Key concepts

Worked examples

 # canario.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-stable
spec:
  replicas: 4                         # 4 of 5 Pods: about 80% of the traffic
  selector:
    matchLabels:
      app: myapp
      track: stable
  template:
    metadata:
      labels:
        app: myapp                    # common label, used by the Service
        track: stable                 # label that distinguishes the version
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-text=v1 estavel"]
          ports:
            - containerPort: 5678
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-canary
spec:
  replicas: 1                         # 1 of 5 Pods: about 20% of the traffic
  selector:
    matchLabels:
      app: myapp
      track: canary
  template:
    metadata:
      labels:
        app: myapp
        track: canary
    spec:
      containers:
        - name: app
          image: hashicorp/http-echo:1.0
          args: ["-text=v2 canario"]
          ports:
            - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp                        # without the "track" label: the Service sees both Deployments
  ports:
    - port: 80
      targetPort: 5678

Essential commands for this stage

Hands-on

1. Apply canario.yaml and run kubectl get endpoints myapp. Observe the 5 endpoints (4 stable and 1 canary).
2. Run the 20-request loop with the cliente Pod. Observe the approximate count of 80% for v1 estavel and 20% for v2 canario. Run it again and observe that the numbers change: the split is statistical, not exact.
3. Abort the canary with kubectl scale deployment myapp-canary --replicas=0 and repeat the loop. Observe that only v1 estavel answers.
4. Reflect on the limit of the technique: for a 1% canary, how many replicas would be needed? That is the reason service meshes and Argo Rollouts with weight-based routing exist.
5. Clean up: kubectl delete -f canario.yaml.

Stage 07 node scaling

Node scalability with Karpenter

The problem

The concept

Karpenter (created by AWS, open source, with providers for other clouds) provisions nodes on demand, sized for the Pods that are pending, without fixed groups. It reacts in seconds to a few minutes, chooses the most suitable instance type (including Spot) and removes idle nodes on its own. Companies report savings of up to 30% in compute cost, a figure that depends on the environment.

The provisioning flow:

1. Pods become unschedulable: the scheduler did not find a node (Pending, with the event FailedScheduling).
2. Karpenter observes these Pods and adds up their resource requests.
3. It calculates which instance type best accommodates the set, within the rules of the NodePool.
4. It provisions the node (through a NodeClaim), and the Pods are scheduled on it.

The inverse flow, consolidation: Karpenter identifies underused nodes, moves the Pods to other nodes and removes the node that is left over, or swaps an expensive instance for a cheaper one. Consolidation waits a configurable time (consolidateAfter) before acting, so as not to react to short oscillations.

Configuration in two objects:

• NodePool: defines the rules for the nodes Karpenter may launch (allowed instance types, zones, Spot or On-Demand, resource limits). Older versions called this object a Provisioner. The recommendation is to restrict as little as possible, to give freedom for optimization.
• EC2NodeClass (on AWS): defines what is provider-specific (AMI, subnets, security groups, the nodes' IAM role). The NodePool points to it through nodeClassRef.

The underlying problem is multidimensional bin packing (each Pod is an item with a vector of CPU and memory, and each node is a container), which is NP-hard. Karpenter uses a greedy heuristic: it discards types that do not fit (no GPU when the Pod asks for a GPU, wrong zone), sorts by cost and suitability and looks for a solution with little leftover, similar to First-Fit Decreasing. The key difference from the Cluster Autoscaler is right-sizing: a small node for a small Pod, a large node when large Pods arrive, mixing types as needed.

Costs and Spot. Spot instances cost much less than On-Demand (on the order of 70% to 90% discount, depending on the type and the moment), but the provider may take them back with short notice (2 minutes on AWS). Karpenter handles this with proactive interruption: it detects the notice, marks the node as terminating, applies a taint and drains the Pods to another node before the cutoff. Two important precisions:

• For voluntary disruptions (consolidation, drift, expiration), Karpenter respects PodDisruptionBudgets and does not bring down Pods beyond what the budget allows.
• The Spot interruption is involuntary: the provider will take the machine back anyway, and the 2-minute deadline may not be enough to drain everything. That is why the PDB reduces the risk but does not eliminate it.

Good practices: a PDB to limit how many Pods leave together, Topology Spread to distribute replicas across nodes and zones, and separate NodePools for critical workloads (On-Demand) and interruption-tolerant workloads (Spot).

Reported cases. The source material mentions Mercado Livre (use of Karpenter to mix instance types and use Spot and reserved capacity, eliminating dozens of node groups) and Tinybird (a cut of about 20% of the AWS bill, 90% in the CI environment, by consolidating everything into two NodePools: one critical On-Demand and one stateless 100% Spot). The numbers come from the original report.

Key concepts

Worked examples

 # nodepool.yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]      # allows both; Karpenter prioritizes Spot when possible
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
      nodeClassRef:                          # in v1 the nodeClassRef needs group, kind and name
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: "1000"                              # ceiling of the combined CPU of the nodes in this NodePool
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # consolidates empty or underused nodes
    consolidateAfter: 30s                    # waits 30s before acting
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: "KarpenterNodeRole-meu-cluster"      # the nodes' IAM role (placeholder)
  amiSelectorTerms:
    - alias: al2023@latest                   # Amazon Linux 2023 AMI
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: meu-cluster  # discovers the subnets by tag
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: meu-cluster

Essential commands for this stage

Hands-on

1. Apply pod-grande.yaml on Minikube and run kubectl get pods. Observe the Pod in Pending.
2. Run kubectl describe pod -l app=grande and observe in Events the message FailedScheduling with Insufficient cpu. This is the signal Karpenter uses to decide to create a node.
3. Run kubectl get nodes -L karpenter.sh/capacity-type and observe the empty column: no node was provisioned by Karpenter, which confirms it is not installed.
4. Read the NodePool from the example and answer: what happens if limits.cpu is reached? (Answer: Karpenter does not create new nodes in that NodePool and the Pods stay Pending.)
5. Clean up: kubectl delete -f pod-grande.yaml.

Stage 08 event-driven scaling

Application scalability with KEDA

The problem

The concept

KEDA (Kubernetes Event-driven Autoscaler) scales based on external events: queues (RabbitMQ, SQS, Kafka, Redis), Prometheus metrics, cron, among others, and supports scale-to-zero.

The central object is the ScaledObject. It points to a target Deployment and defines one or more triggers. How it works in practice:

1. KEDA queries the trigger's source every pollingInterval seconds.
2. Empty queue: keeps the Deployment at 0 replicas (if minReplicaCount: 0).
3. An event arrives: KEDA activates the Deployment (from 0 to 1).
4. From 1 on, what calculates the number of replicas is an HPA that KEDA itself creates: for example, with value: "10" in QueueLength mode, 40 messages result in 4 Pods.
5. The queue becomes empty and stays that way for cooldownPeriod seconds: KEDA takes the Deployment back to 0.

Three points of precision. First, the HPA alone cannot go to zero (the minimum is 1), so the activation from 0 to 1 and the return to 0 are KEDA's responsibilities. Second, the cooldownPeriod (default 300 seconds) counts the time during which the trigger is inactive (empty queue) and only applies to the move to zero. Third, the difference between the two ways of scaling down: when the trigger becomes inactive, KEDA zeroes the Deployment directly, from any number of replicas (from 4 to 0, for example), at the end of the cooldownPeriod; the HPA's scale-down stabilization window (default 5 minutes, adjustable in the ScaledObject under advanced.horizontalPodAutoscalerConfig.behavior), on the other hand, applies to reductions while the queue is still active, such as going from 5 to 3 replicas with the queue shrinking but still holding items.

Queueing theory. Calling λ the arrival rate of events and μ the rate at which each Pod processes, with c Pods the capacity is c·μ. The stability condition is λ < c·μ: if arrivals are sustainedly greater than the capacity, the queue grows without limit and no autoscaling solves it. Little's Law (L = λ·W) relates the average number of items in the system L, the arrival rate λ and the average time in the system W: if the backlog grows with constant λ, W is increasing, that is, there are not enough Pods.

Oscillation. A control challenge is to avoid the up and down of replicas, caused by feedback delay (a new Pod takes time to become ready) and by fixed thresholds. Mechanisms that the HPA and KEDA offer: a tolerance (the default HPA only reacts when the metric exceeds about 110% or falls below about 90% of the target, a margin of 10%), stabilization windows and the cooldownPeriod. A general control concept is the dead band (scale up at one threshold and down at another, lower one); it illustrates the idea of hysteresis, but it is not a direct configuration of the HPA or KEDA, which use tolerance and windows. Since queues absorb bursts, a temporary backlog is accepted while the new Pods come up.

KEDA is provider-independent (AKS, EKS, GKE, on-premises) and complements the HPA, without replacing it. A typical case: order workers scaled by the SQS or Service Bus queue during flash sales, returning to zero in the small hours. For event sources that KEDA does not support, there are external scalers via gRPC.

Key concepts

Worked examples

 # scaledobject-rabbitmq.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker               # target Deployment
  minReplicaCount: 0           # allows scale-to-zero
  maxReplicaCount: 5
  cooldownPeriod: 60           # queue empty for 60s before going to zero
  pollingInterval: 15          # queries the queue every 15s
  triggers:
    - type: rabbitmq
      metadata:
        queueName: pedidos
        protocol: amqp
        mode: QueueLength      # scales by the queue size
        value: "10"            # one Pod for every 10 messages
        hostFromEnv: RABBITMQ_URL   # variable with the broker's URL, defined in the target Deployment's container

Essential commands for this stage

Hands-on

1. Install KEDA with Helm and apply keda-lab.yaml. Run kubectl get scaledobject and observe READY True and ACTIVE False (no work).
2. Run kubectl get deployment worker and observe 0/0: the worker went to zero, something the HPA alone does not do.
3. In a second terminal, run kubectl get deployment worker -w. In the first, push 20 items with LPUSH. Observe the number of replicas rise to 4, and check with kubectl get hpa the HPA created by KEDA.
4. Delete the queue with redis-cli DEL fila and observe the Deployment go from 4 replicas straight to 0 after about 30 seconds (the exercise's cooldownPeriod). This is faster than the HPA's 5 minutes of stabilization because, with the queue empty, the trigger becomes inactive and KEDA zeroes the Deployment without going through the gradual descent; the HPA's window would only apply if the queue were shrinking but still holding items.
5. Push 100 items and observe the limit maxReplicaCount: 5: even with the queue asking for 20 Pods, KEDA respects the ceiling.
6. Clean up: kubectl delete -f keda-lab.yaml and helm uninstall keda -n keda.

Stage 09 rbac + identity + tls

Cluster security

The problem

The concept

Identity: ServiceAccounts. Each application should have its own ServiceAccount, never the namespace's default. The ServiceAccount gives the Pod a token to talk to the API Server. Separate identities per application make it possible to contain the damage when a Pod is compromised. If the application never talks to the API Server, turn off even the mounting of the token with automountServiceAccountToken: false.

Authorization: RBAC. Permissions are granted by Roles (restricted to a namespace) or ClusterRoles (cluster-wide) and tied to subjects by RoleBindings (or ClusterRoleBindings). The model is a graph: subject, binding, role and permission. Least-privilege rules:

• Prefer Role (namespaced) over ClusterRole.
• Grant only the necessary verbs (get, list) and only the necessary resources.
• Avoid the wildcard * in verbs and resources.

Kubernetes RBAC is additive: only rules that grant permission exist, there are no denial rules, and what was not granted is denied. This makes it auditable: it is possible to list deterministically who can do what. ABAC (attribute-based) is more expressive, but opaque, and it is a legacy mechanism, rarely used. For conditional policies ("deny a Pod without limits", "only accept images from a certain registry"), an admission mechanism such as OPA/Gatekeeper is used, with the Rego language (policy as code), or Kubernetes' native admission policy features. The more expressive the policy, the harder it is to prove its properties.

Federated identity with the cloud. Applications need to access S3, buckets and secret vaults without embedding static keys in the container. The solution is to federate the Kubernetes ServiceAccount with a cloud identity through OIDC, receiving short-lived credentials:

• IRSA (IAM Roles for Service Accounts) on EKS: the ServiceAccount receives an annotation with the ARN of an IAM Role. Each application has its own Role, with the minimum necessary. (EKS also offers EKS Pod Identity as a more recent alternative.)
• Workload Identity on GKE and AKS: the ServiceAccount is associated with a cloud identity (a service account on GKE, a managed identity on AKS).

This eliminates static secrets and restricts the credential to the Pod's context. The same principle holds for IAM: the attack known as SCARLETEEL (2023, documented by Sysdig) used credentials obtained from containers to move around cloud environments, and the episode reinforces the lesson of keeping IAM permissions to a minimum and not leaving static secrets in Pods.

Automated TLS: cert-manager. Managing certificates by hand does not scale: they expire and become inconsistent. cert-manager automates the life cycle through CRDs: Issuer and ClusterIssuer (who issues: self-signed, internal CA, ACME/Let's Encrypt, Vault) and Certificate (what to issue, with name, SANs, duration and renewBefore for early renewal). It generates the request, obtains the signature, fills in a Secret with tls.crt and tls.key and renews on its own. The certificate is X.509, part of a chain of trust that ends at a root CA. Kubernetes itself uses more than one CA (etcd, API server, front-proxy) to compartmentalize: if one leaks, the others remain intact.

Secret rotation. Secrets should be changed to reduce the exposure window. An instant swap is impractical in distributed systems (there is no global lock), so a coexistence period is used: the new credential and the old one are valid at the same time until everyone has migrated, and then the old one is invalidated. Dynamic secrets (Vault, short-lived tokens) are already ephemeral and almost dispense with rotation. In the rotation of the etcd encryption key, all API servers need to know the new key *before* it starts being used for writing, otherwise an API server that does not know it cannot read the data written with it.

Key concepts

Worked examples

 # rbac-analytics.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: analytics-sa
  namespace: analytics
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/S3Reader   # IRSA: only works on EKS; fictitious ARN
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role                             # restricted to the namespace, it is not a ClusterRole
metadata:
  name: analytics-read
  namespace: analytics
rules:
  - apiGroups: [""]                    # the API's "core" group
    resources: ["pods"]
    verbs: ["get", "list"]             # read-only, no "*"
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: bind-analytics-read
  namespace: analytics
roleRef:                               # which Role
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: analytics-read
subjects:                              # for whom
  - kind: ServiceAccount
    name: analytics-sa
    namespace: analytics

Essential commands for this stage

Hands-on

1. Create the RBAC test environment: kubectl create namespace lab, kubectl create serviceaccount leitor -n lab, kubectl create role ler-pods --verb=get,list --resource=pods -n lab and kubectl create rolebinding bind-leitor --role=ler-pods --serviceaccount=lab:leitor -n lab.
2. Test with kubectl auth can-i list pods -n lab --as=system:serviceaccount:lab:leitor. Observe yes.
3. Test delete pods and get secrets (which were not granted). Observe no on both. Also test list pods -n default: observe no, because the Role only applies in the namespace lab.
4. Apply rbac-analytics.yaml (after creating the namespace analytics) and run kubectl create token analytics-sa -n analytics. Observe that the token is generated on the spot, with limited validity, instead of a permanent credential.
5. Install cert-manager, apply pki.yaml and run kubectl get certificate. Observe READY True on both certificates and check with openssl the issuer of the service's certificate (openssl must be installed on your machine; without it, use kubectl describe certificate service-a-cert). This demonstrates automatic issuance; renewal happens on its own, at the renewBefore interval ahead of expiry.
6. Read the IRSA example without running it and list what would be missing for it to work (an EKS cluster, an OIDC provider, an IAM Role with the trust relationship with the ServiceAccount).
7. Clean up: kubectl delete namespace lab analytics, kubectl delete -f pki.yaml, kubectl delete secret ca-interna-tls service-a-tls (cert-manager does not delete the Secrets it created by default) and helm uninstall cert-manager -n cert-manager.

Stage 10 bonus · podinfo in production

Bonus, podinfo from the basics module goes to production

The problem

The concept

This stage turns the manifests from Stage 09 of the basics guide into a Helm chart and reinforces the Deployment with the pieces from this guide. Each piece comes from the stage that taught it:

• Helm (Stage 04): the ConfigMap, the Service and the HPA go into the chart without relevant changes, as templates. The same chart serves dev, staging and production, changing only the values file.
• Minimal ServiceAccount (Stage 09): podinfo gets a ServiceAccount of its own. It is an HTTP server that never calls the API Server, so it receives no Role or RoleBinding, and the token is not even mounted in the Pod (automountServiceAccountToken: false). It is least privilege taken to the limit: no credential where none is needed.
• No CPU limit (Stage 01): podinfo is latency-sensitive, so the chart keeps CPU and memory requests and a memory-only limit, to avoid throttling. The Pod is Burstable.
• Topology Spread (Stage 02): spreads the replicas across nodes with maxSkew: 1, so that the loss of one node does not take all the replicas. In multi-node production, use DoNotSchedule; on Minikube, with a single node, ScheduleAnyway, because DoNotSchedule would leave the replicas beyond the first in Pending forever.
• PodDisruptionBudget (Stage 02): guarantees a floor of replicas during voluntary disruptions (drain, node upgrade, Karpenter consolidation). As seen in Stage 02, it does not protect against eviction under resource pressure.
• Tuned rollout (Stage 03): maxUnavailable: 0 to never reduce capacity, maxSurge: 1 to create one extra Pod at a time, minReadySeconds and the change-cause annotation to record each swap.
• Canary (Stage 06): a second release of the same chart, with 1 replica and the new version, receives a fraction of the traffic of the same Service (the native technique by replica ratio). The promotion is a helm upgrade --atomic.

Why the chart needs to distinguish the canary. Two releases of the same chart cannot create objects with the same name (Helm refuses, because the object belongs to another release). That is why the Deployment's name comes from .Release.Name, and Service, ConfigMap, HPA, ServiceAccount and PDB are only created by the stable release (canary: false). The Pods distinguish the version with a track label (stable or canary), used in the selector of each Deployment, so that two Deployments do not fight over the same Pods. The Service selects only by app: podinfo, so it sees both.

What is left out, and why:

• Karpenter and IRSA or Workload Identity are cloud concepts, with no effect on a single-node Minikube; they stay read-only in Stages 07 and 09.
• cert-manager: with no ingress with TLS in this lab, there is nothing to issue or renew.
• KEDA: the HPA that already exists covers podinfo's profile, an HTTP API that should not go to zero; scale-to-zero by queue makes sense for queue workers.
• Blue/Green: for a single public Service, one promotion strategy is chosen. The canary, progressive and metric-driven, fits better with a service that already exposes /healthz and /readyz.
• PVC: stays out, for the same reason as in the basics guide (a ReadWriteOnce volume shared among several replicas).
• podAntiAffinity: the chart follows the advice of Stage 02 and uses Topology Spread instead of affinity between Pods.

Key concepts

Worked examples

 # Chart.yaml
apiVersion: v2
name: podinfo
description: podinfo em formato de produção
type: application
version: 0.1.0                     # chart version
appVersion: "6.14.1"               # application version

Essential commands for this stage

Hands-on

1. Assemble the folder podinfo-chart/ with Chart.yaml, values.yaml and the six files of templates/. Run helm lint ./podinfo-chart and helm template podinfo ./podinfo-chart and observe the rendered YAML, with the Deployment, the Service, the HPA, the ServiceAccount and the PDB.
2. Run helm template podinfo-canary ./podinfo-chart --set canary=true and observe that only the Deployment appears as an object (named podinfo-canary and with track: canary); the other files appear empty or with only the comment from the first line, under # Source:. This is what avoids name collisions between the two releases.
3. Check the tags with docker manifest inspect stefanprodan/podinfo:6.14.1 and docker manifest inspect stefanprodan/podinfo:6.15.0. With both confirmed, install production: helm install podinfo ./podinfo-chart -n podinfo --create-namespace, and confirm 3 Running Pods with kubectl get pods -n podinfo -o wide.
4. Run the three kubectl auth can-i calls for podinfo-sa. Observe no on all three: the app has no reason to call the API Server, so it received no credential at all.
5. Run kubectl get pdb podinfo -n podinfo and observe ALLOWED DISRUPTIONS 1 (3 healthy Pods, minimum of 2).
6. Bring up the canary with helm install podinfo-canary ... --set canary=true --set replicaCount=1 --set image.tag=6.15.0 and observe in kubectl get endpoints podinfo -n podinfo the 4 IPs, one of them with the new version serving real traffic.
7. Promote with helm upgrade podinfo ... --set image.tag=6.15.0 --atomic, remove the canary with helm uninstall podinfo-canary -n podinfo and confirm in kubectl rollout history deployment/podinfo -n podinfo that the swap was recorded.
8. Clean up both releases: the canary already left in step 7, so run helm uninstall podinfo -n podinfo and kubectl delete namespace podinfo.

Stage 11 · review

Self-assessment checklist

Next steps: with all items checked, redo Stage 10 from scratch without consulting the guide, this time on a multi-node Minikube profile (like the one in Stage 02) and with whenUnsatisfiable: DoNotSchedule, and check the distribution of the replicas with kubectl get pods -o wide. Then, if you have access to a test cloud account, run the snippets that were read-only here: Karpenter's NodePool, IRSA or Workload Identity and the exact-weight canary with Istio or Argo Rollouts.