Server scalability
through the F1 lens.

Lab

1. Baseline: one instance, capacity 2, 100 ms per request.

docker run -d --name base -p 8081:8080 lab-app
curl -s -o /dev/null -w "latência de uma requisição: %{time_total}s\n" localhost:8081/
carga http://localhost:8081/

Expected output (approximate values, your machine varies):

latência de uma requisição: 0.10xxxxs
real    0m5.2s

The math: 2 requests every 100 ms give 20 per second; 100 requests take about 5 seconds.

2. Optimization: same machine, half the work.

docker run -d --name otim -p 8082:8080 -e TRABALHO_MS=50 lab-app
curl -s -o /dev/null -w "latência: %{time_total}s\n" localhost:8082/
carga http://localhost:8082/

Expected: latency near 0.05s and total time near 2.5 s.

3. Vertical scaling: a "bigger server" (capacity 4).

docker run -d --name vert -p 8083:8080 -e CAPACIDADE=4 lab-app
curl -s -o /dev/null -w "latência: %{time_total}s\n" localhost:8083/
carga http://localhost:8083/

Expected: latency stays near 0.10s; total time near 2.5 s.

4. Horizontal scaling: three small instances, with a hand-made "balancer".

docker run -d --name h1 -p 8084:8080 lab-app
docker run -d --name h2 -p 8085:8080 lab-app
docker run -d --name h3 -p 8086:8080 lab-app
 
# The request number modulo 3 picks port 8084, 8085 or 8086
time (seq 1 100 | xargs -P 20 -I{} sh -c 'curl -s -o /dev/null http://localhost:$((8084 + {} % 3))/')

Expected: total time near 1.7 s (3 instances × 20 per second = 60 per second).

5. What to observe.

Scaling (vertical or horizontal) did not change the latency of an isolated request, only the throughput. Optimizing changed both. This is exactly the distinction between performance and scalability.

6. The single point of failure, live.

docker stop base
curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 localhost:8081/
 
docker stop h1
for p in 8084 8085 8086; do curl -s -o /dev/null -w "%{http_code} " --max-time 2 localhost:$p/; done; echo

Expected: 000 on the first call (connection refused: the SPOF went down) and 000 200 200 on the second. With the fleet, only a third fails. But whoever decides where to send the request is still your sh -c, which does not know h1 went down. In the next stage, the load balancer decides.

Cleanup:

docker rm -f base otim vert h1 h2 h3

Exercises

1. Scale or optimize? For each situation, say which lever to use first and why: (a) a page runs 400 SQL queries in a loop; (b) the Black Friday campaign site will receive 20 times the normal traffic in 3 hours; (c) an internal system with 50 employees is slow on Monday mornings. *Expected answer:* (a) optimize, because the problem is the query and scaling would only multiply the waste; (b) scale horizontally, preferably with scheduled or metric-based autoscaling; (c) measure first, probably optimize, and avoid autoscaling as it is overkill in cost and complexity.
2. Write the three rules. For a store with a predictable peak at 20:00, write a metric-based rule, a scheduled one and describe what a predictive one would do differently. *Expected answer:* (metric) "if the fleet's average CPU exceeds 70% for more than 5 minutes, add a server; if it drops below 30% for 10 minutes, remove one"; (scheduled) "every day at 19:50, scale up to the number of servers the peak requires, and at 22:00 scale down to the minimum"; (predictive) a Machine Learning model trained on the history learns the shape of the peak (for example, that Friday is stronger than Tuesday) and raises capacity before it begins, without depending on a hand-written time. The underlying difference: the metric rule reacts after the load has risen; the scheduled and predictive ones anticipate.
3. Triangle. You need to cut the database latency in half. Name one solution that sacrifices cost and another that sacrifices complexity. *Expected answer:* sacrificing cost, top-end hardware for the database (vertical scaling: more RAM, faster disks), simple to operate but expensive, with a ceiling and a single point of failure; sacrificing complexity, one or more cache layers (for example, Redis) in front of the database, cheap per unit but with new problems (invalidation, stale data). Before either, it is worth checking whether a better query or index already solves it, because optimization costs little on all three vertices.

Lab

1. Round-robin. 30 requests, counting who answered:

for i in $(seq 1 30); do curl -s localhost:8080/; done | awk '{print $1}' | sort | uniq -c

Expected (the IDs change):

     10 3f2a1b9c0d1e
     10 7c8d9e0f1a2b
     10 a1b2c3d4e5f6

Thanks to the zone directive, the Nginx worker processes share the same round-robin counter, which is why the split comes out exact. Without it, each worker would have its own counter and the distribution could come out uneven (for example, 12 for one and 9 for each of the others).

2. Failover. Take down one replica and keep calling:

docker stop $(docker compose ps -q app | head -n 1)
for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/; done | sort | uniq -c

Expected: 20 200. No failure reached the client. The first request that lands on the stopped replica may take about 1 second (the proxy_connect_timeout) before Nginx tries another server. Compare with Stage 01, where 000 appeared.

Bring the replica back and refresh Nginx, which resolves names only at startup and so may hold an outdated IP:

docker compose up -d
docker compose exec lb nginx -s reload

3. Algorithms. Switch to least_conn and then to ip_hash. (On macOS, use gsed in place of sed, or edit nginx/default.conf by hand: insert the line least_conn; right below upstream azul { and, afterwards, replace it with ip_hash;.)

# Inserts "least_conn;" right after the opening of the upstream azul block
sed -i 's/upstream azul {/upstream azul {\n    least_conn;/' nginx/default.conf
docker compose exec lb nginx -t && docker compose exec lb nginx -s reload
 
# Now switches to ip_hash
sed -i 's/least_conn;/ip_hash;/' nginx/default.conf
docker compose exec lb nginx -t && docker compose exec lb nginx -s reload
for i in $(seq 1 10); do curl -s localhost:8080/; done | awk '{print $1}' | sort | uniq -c

Expected with ip_hash: a single line with 10 and a single server. Your client always has the same IP (the Docker gateway's) and so always lands on the same server: this is session affinity, and also the imbalance that happens when many people share an IP. Remove the line to go back to round-robin:

sed -i '/ip_hash;/d' nginx/default.conf
docker compose exec lb nginx -s reload

4. Canary at 10%. Replace the contents of nginx/default.conf with:

upstream azul {
    zone azul 64k;                      # balancing state shared among the workers
    server app:8080 max_fails=1 fail_timeout=10s;
}
 
upstream verde {
    server verde:8080;
}
 
# Draws the destination by each request's unique ID: 10% go to "verde", the rest to "azul"
split_clients "${request_id}" $destino {
    10% verde;
    *   azul;
}
 
server {
    listen 80;
    location / {
        proxy_pass http://$destino;     # the group name comes from the drawn variable
        proxy_connect_timeout 1s;
        proxy_next_upstream error timeout;
    }
}

docker compose exec lb nginx -t && docker compose exec lb nginx -s reload
seq 1 200 | xargs -P 10 -I{} curl -s localhost:8080/ | awk '{print $2}' | sort | uniq -c

Expected, approximately: 180 v1 and 20 v2 (the draw varies a little). If the v2 metrics got worse, only that 10% would feel it.

5. Blue/Green: the switch and the rollback. (On macOS, use gsed, or edit the file by hand: replace 10% with 100% for the switch and delete the verde line for the rollback.)

# Switch: 100% to the new version
sed -i 's/10% verde;/100% verde;/' nginx/default.conf
docker compose exec lb nginx -s reload
for i in 1 2 3 4 5; do curl -s localhost:8080/ | awk '{print $2}'; done
# Expected: v2 five times
 
# Rollback: removes the verde rule and sends everything back to azul
sed -i '/% verde;/d' nginx/default.conf
docker compose exec lb nginx -s reload
for i in 1 2 3 4 5; do curl -s localhost:8080/ | awk '{print $2}'; done
# Expected: v1 five times

nginx -s reload is graceful: the old processes finish the connections in progress before exiting. This is the idea of connection draining.

Exercises

Exercises 1 and 2 change nginx/default.conf with the stack running. The file is in the canary/rollback format (with split_clients), and it is its location / block that you replace in exercise 1. After editing, run docker compose exec lb nginx -t && docker compose exec lb nginx -s reload.

1. L7 routing. Replace the location / block with two: location /api/ with proxy_pass http://verde; and location / with proxy_pass http://azul;. Call curl -s localhost:8080/api/pagamentos and curl -s localhost:8080/blog. Which version answers each one? Why is this only possible at L7? *Expected answer:* /api/pagamentos answers with v2 (the verde group) and /blog with v1 (the azul group). Only L7 can do it, because the /api/... path is inside the HTTP request, and L4 sees only IP and port.
2. Weights. What would be the effect of server app:8080 weight=3; next to another server with weight 1, and in what real scenario would you use it? *Expected answer:* the weight-3 server would receive about 3 out of every 4 requests (ratio 3 to 1). It is Weighted Round-robin, useful when the machines are unequal and a more powerful server should receive more load than a smaller one.
3. Ingress (reading). In the YAML of the second worked example, which service serves /api/pedidos and which serves /? *Expected answer:* /api/pedidos goes to the Service api; / goes to web. It is the same reasoning as the Nginx location blocks.

Cleanup (when you finish the lab and the exercises):

docker compose down -v
cd ..

Lab

1. The problem: stateful cart.

export MODO=memoria
docker compose up -d --build
 
# Three items, one per request; the LB distributes among the replicas
for i in 1 2 3; do curl -s -X POST "localhost:8080/carrinho?user=ana&item=item$i"; done
# Now reads the cart three times
for i in 1 2 3; do curl -s "localhost:8080/carrinho?user=ana"; done

Expected (IDs and order vary):

{"servidor": "aaa111", "modo": "memoria", "user": "ana", "itens": ["item1"]}
{"servidor": "bbb222", "modo": "memoria", "user": "ana", "itens": ["item2"]}
{"servidor": "ccc333", "modo": "memoria", "user": "ana", "itens": ["item3"]}

Each response has only one item. Ana's cart is spread across three memories and no replica sees it whole. This is the problem of state tied to the server. (The zone in the upstream makes the Nginx workers share the round-robin, so the six requests go through the three replicas in order and each read lands on the replica that received the corresponding item.)

2. The solution: state in Redis.

docker compose down
export MODO=redis
docker compose up -d
for i in 1 2 3; do curl -s -X POST "localhost:8080/carrinho?user=ana&item=item$i"; done
for i in 1 2 3; do curl -s "localhost:8080/carrinho?user=ana"; done
docker compose exec redis redis-cli LRANGE carrinho:ana 0 -1

Expected: the three responses show "itens": ["item1", "item2", "item3"], coming from different servers, and redis-cli lists:

1) "item1"
2) "item2"
3) "item3"

3. Failover without losing the cart.

docker stop $(docker compose ps -q app | head -n 1)
curl -s "localhost:8080/carrinho?user=ana"

Expected: the cart is still complete. It may take up to about 1 second if the request lands on the stopped replica: open-source Nginx only notices the failure when it tries, and proxy_connect_timeout 1s limits the wait before it moves to another replica. Bring the replica back and refresh Nginx, which resolves names only at startup (the replica may come back with another IP):

docker compose up -d
docker compose exec lb nginx -s reload

4. Queue and workers. The order returns fast and the heavy work happens later:

# Turns on 1 worker (MODO stays exported, so the API is not recreated in another mode)
docker compose up -d --scale worker=1
 
# Enqueues 12 orders; each response arrives immediately with 202
for i in $(seq 1 12); do curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" -X POST "localhost:8080/pedido?id=$i"; done
 
# Queue size
docker compose exec redis redis-cli LLEN fila:pedidos

Expected: twelve lines like 202 0.00xs and an (integer) N with N between 1 and 11, because the worker is already consuming. Watch the worker working (Ctrl+C to exit):

docker compose logs -f worker

Expected output (shortened):

lab-etapa-03-worker-1  | [9f8e7d6c5b4a] processando pedido 1
lab-etapa-03-worker-1  | [9f8e7d6c5b4a] pedido 1 concluido

5. Scale the workers by queue size. Wait for the previous step's queue to empty (LLEN equal to 0, about 12 s) so the measurement does not add old orders. Then measure the time to empty the queue with 1 worker and with 3:

# With 1 worker (12 orders of 1 s: near 12 s)
for i in $(seq 1 12); do curl -s -o /dev/null -X POST "localhost:8080/pedido?id=$i"; done
time { while [ "$(docker compose exec -T redis redis-cli LLEN fila:pedidos | tr -d '\r')" != "0" ]; do sleep 0.5; done; }
 
# With 3 workers (near 4 s)
docker compose up -d --scale worker=3
for i in $(seq 1 12); do curl -s -o /dev/null -X POST "localhost:8080/pedido?id=$i"; done
time { while [ "$(docker compose exec -T redis redis-cli LLEN fila:pedidos | tr -d '\r')" != "0" ]; do sleep 0.5; done; }

The workers scaled independently of the API, and it was the queue size that justified the scaling. This is the idea of KEDA.

Exercises

Exercises 1 and 3 need the stack running: export MODO=redis in exercise 1 and export MODO=memoria in exercise 3. To switch modes, run docker compose down, export the new value and docker compose up -d.

1. KEDA by hand. Suppose each worker processes 1 order per second and you want to empty the queue within 4 seconds. Write the formula for the number of workers as a function of LLEN (round up) and test it with --scale. *Expected answer:* workers = ceil(LLEN / 4), because each worker handles 4 orders in 4 seconds. In Bash, n=$(( (LLEN + 3) / 4 )). With LLEN equal to 12, that is 3 workers (docker compose up -d --scale worker=3) and the queue empties in about 4 seconds. This is the logic KEDA automates.
2. Retry with exponential backoff. Run the script below against a port where nothing is listening. During the wait, in another terminal, start docker run -d --rm -p 9999:8080 lab-app and watch the script succeed.

   for tentativa in 1 2 3 4 5; do
     # --fail treats an HTTP error as a failure; --max-time 2 limits the wait of each attempt
     if curl -sf --max-time 2 -o /dev/null http://localhost:9999/health; then
       echo "ok na tentativa $tentativa"; break
     fi
     espera=$((2 ** (tentativa - 1)))   # waits 1, 2, 4, 8, 16 seconds
     echo "tentativa $tentativa falhou, esperando ${espera}s"
     sleep "$espera"
   done

   *Expected answer:* lines like tentativa 1 falhou, esperando 1s, tentativa 2 falhou, esperando 2s, and so on, until ok na tentativa N appears on the first attempt after the container starts. The interval doubles with each failure, so the script keeps trying without hammering the service. If you do not start the container, all five attempts fail and the script ends without the ok line. At the end, stop the test container with docker stop $(docker ps -q --filter publish=9999).
3. Sticky session, the patch. With the stack in MODO=memoria, add ip_hash; to the upstream backend in nginx/default.conf (right below the line upstream backend {), run docker compose exec lb nginx -s reload and repeat the cart test from step 1 with a new user (for example, user=bia). Does it work? Now take down the replica that served (the "servidor" field of the response is the container's short ID: docker stop ID) and read the cart again. What happened, and why does this limit resilience? *Expected answer:* with ip_hash, all requests from your client (always the same IP) land on the same replica, so the three reads show ["item1", "item2", "item3"], always with the same "servidor": the patch works as long as that replica lives. After docker stop, the next read is diverted to another replica (possibly with about 1 s of waiting) and comes back with "itens": []. The cart lived in the memory of the replica that went down and died with it. Sticky session does not take the state out of the server, it only hides the problem: the failure of one instance still loses the data of the clients tied to it, and the load is distributed by IP, not by capacity. When done, delete the ip_hash; line, run docker compose up -d to recreate the replica and docker compose exec lb nginx -s reload.
4. ReplicaSet (reading). In the YAML of the second worked example, map each field to Compose: which is the replicas, which is the image, and what has no direct equivalent? *Expected answer:* spec.replicas: 3 corresponds to deploy.replicas: 3; the image inside template.spec.containers corresponds to the service's image:; the whole template is the mold, like the definition of a service in Compose; metadata.name plays the role of the service name; containerPort: 80 documents the port, like the Dockerfile's EXPOSE. The selector with matchLabels has no equivalent that you write in Compose: there the replicas are grouped by the service name, while in Kubernetes the ReplicaSet finds "its" Pods by the label app: carrinho.

Cleanup (when you finish the lab and the exercises):

docker compose down -v
unset MODO
cd ..

Lab

1. Performance detective at the origin. Inspect the headers:

curl -sI http://localhost:8082/estatico/app.js

Expected (shortened):

HTTP/1.1 200 OK
...
ETag: "6650a1b2-14"
Cache-Control: public, max-age=10

2. The 304 happening. Get the ETag and ask "did it change?":

ETAG=$(curl -sI http://localhost:8082/estatico/app.js | tr -d '\r' | awk 'tolower($1)=="etag:" {print $2}')
curl -sI -H "If-None-Match: $ETAG" http://localhost:8082/estatico/app.js

Expected: HTTP/1.1 304 Not Modified, with no body. The server confirmed the copy is still valid without resending the file.

3. Miss and hit at the edge.

for i in 1 2 3; do curl -s -D - -o /dev/null http://localhost:8081/estatico/app.js | tr -d '\r' | grep -i '^x-cache'; done

Expected:

X-Cache: MISS
X-Cache: HIT
X-Cache: HIT

4. Cache Hit Ratio. Twenty requests and the count:

for i in $(seq 1 20); do curl -s -D - -o /dev/null http://localhost:8081/estatico/app.js | tr -d '\r' | grep -i '^x-cache'; done | sort | uniq -c

Expected: 20 X-Cache: HIT if you ran it right after step 3 (within the 10 s TTL). If more than 10 s have passed, the result is 1 X-Cache: REVALIDATED and 19 X-Cache: HIT: the edge already has the copy, so the first response after the TTL revalidates with the origin, and MISS does not appear (MISS only happens when the edge has nothing stored, as in step 3). Count the REVALIDATED as a trip to the origin: the hit ratio is 20 of 20 (100%) or 19 of 20 (95%). Confirm that the origin was barely bothered:

docker compose logs origem | grep 'app.js' | wc -l

Expected: 4 or 5. The math is this:

• 3 lines from the curl -sI you sent straight to the origin, on port 8082 (one in step 1 and two in step 2);
• 1 line from the edge, the MISS of step 3;
• plus 1 line only if the TTL expired and there was a revalidation (the 304).

What matters: of the 20 requests in this step, the origin saw at most one.

5. Expiry and revalidation. Wait for the TTL to pass and ask again:

sleep 11
curl -s -D - -o /dev/null http://localhost:8081/estatico/app.js | tr -d '\r' | grep -i '^x-cache'
docker compose logs origem | grep 'app.js' | tail -n 3

Expected: X-Cache: REVALIDATED and, in the origin logs, a line with 304. The edge asked, the origin answered "not changed" and the TTL was renewed without transferring the file.

6. A file that changed: expires and downloads again.

echo 'console.log("v2");' > site/estatico/app.js
curl -s http://localhost:8081/estatico/app.js      # still v1, within the TTL
sleep 11
curl -s -D - http://localhost:8081/estatico/app.js | tr -d '\r' | grep -iE '^x-cache|console'

Expected: the first call shows console.log("v1"); (edge copy, still valid). After the TTL, X-Cache: EXPIRED and console.log("v2"); appear. Between the change and the end of the TTL, users saw the old version: this is the staleness window that every TTL accepts. If the first call already shows v2, the TTL from the previous step had expired; repeat the step with a new edit.

7. Simulated invalidation (purge). Open-source Nginx has no purge API; we simulate it by erasing the cache. First fill the cache, then "invalidate":

echo 'console.log("v3");' > site/estatico/app.js
curl -s http://localhost:8081/estatico/app.js        # v2 or v3, depending on the cache state
docker compose exec edge sh -c 'rm -rf /var/cache/nginx/edge/*'
docker compose restart edge
curl -s -D - http://localhost:8081/estatico/app.js | tr -d '\r' | grep -iE '^x-cache|console'

Expected: X-Cache: MISS and console.log("v3");, right away, without waiting for the TTL. A real CDN does this through an API or panel, and that is why invalidating too much costs efficiency: each purge forces a new miss.

Exercises

Exercises 1 and 2 need the stack running. After editing origem/default.conf, run docker compose exec origem nginx -s reload.

1. Cache the API. In origem/default.conf, add add_header Cache-Control "public, max-age=60"; in the location /api/ and reload. Make 10 calls to localhost:8081/api/produtos before and after the change, looking at X-Cache. What changes? What kind of API response is safe to cache this way? *Expected answer:* before, the 10 calls show as MISS, because the response carries no Cache-Control and the edge does not store it (the origin is queried every time); after, the first is MISS and the next 9 are HIT (within the 60 s). It is safe for public GET responses, equal for all users and that change little, like the product catalog; it does not suit personalized responses.
2. When not to cache. Change the value to Cache-Control "private, no-store". Does X-Cache go back to HIT? Why should a logged-in user's data not stay on a shared edge? *Expected answer:* no. After the 60 s copy from the previous exercise expires (or is erased, as in step 7), the calls go back to MISS, because no-store forbids the edge from storing the response. A logged-in user's data on a shared edge could be delivered to another person, and private indicates that only the user's own browser may store it.
3. How much to cache? For each item, choose a TTL and justify it: company logo, versioned JavaScript bundle, product catalog, user's bank balance. *Expected answer (examples):* company logo, long TTL (days), because it almost never changes; versioned JavaScript bundle, very long TTL (months), because the file name changes with each version and the copy is never stale; product catalog, short TTL (seconds to a few minutes), accepting a small staleness; bank balance, do not cache (private, no-store), because it is personal data and changes all the time.
4. Conceptual. Explain, in three sentences, why the CDN's smart DNS makes the user in Lisbon talk to the Paris PoP and not to the server in Brazil. *Expected answer:* when the browser asks for the site's address, the query reaches the CDN's smart DNS. That DNS identifies the user's location by the IP and returns the IP of the nearest PoP (Paris), instead of the origin's IP. The browser then connects to that PoP, which answers from the cache (hit) and only goes to Brazil if it does not have the copy (miss).

Cleanup (when you finish the lab and the exercises):

docker compose down -v
cd ..

Lab

1. Who is who.

docker compose exec primary psql -U postgres -c "SELECT pg_is_in_recovery();"
docker compose exec replica psql -U postgres -c "SELECT pg_is_in_recovery();"

Expected: f on the primary and t on the replica (the replica is in recovery mode, that is, following the primary).

2. Replication is alive (and asynchronous).

docker compose exec primary psql -U postgres -c "SELECT client_addr, state, sync_state FROM pg_stat_replication;"

Expected:

 client_addr |   state   | sync_state
-------------+-----------+------------
 172.x.x.x   | streaming | async

3. Write to the primary, read from the replica.

docker compose exec primary psql -U postgres -c "CREATE TABLE produtos (id int primary key, nome text); INSERT INTO produtos VALUES (1,'Camiseta'),(2,'Caneca');"
sleep 1
docker compose exec replica psql -U postgres -c "SELECT * FROM produtos;"

Expected: both rows appear on the replica.

4. The replica does not accept writes.

docker compose exec replica psql -U postgres -c "INSERT INTO produtos VALUES (9,'Intruso');"

Expected: ERROR: cannot execute INSERT in a read-only transaction. This is why the application needs write splitting.

5. Eventual consistency, live. Pause the application of changes on the replica, write to the primary and compare:

docker compose exec replica psql -U postgres -c "SELECT pg_wal_replay_pause();"
docker compose exec primary psql -U postgres -c "INSERT INTO produtos VALUES (3,'Boné');"
echo "primary:"; docker compose exec primary psql -U postgres -tA -c "SELECT count(*) FROM produtos;"
echo "réplica:"; docker compose exec replica psql -U postgres -tA -c "SELECT count(*) FROM produtos;"

Expected: 3 on the primary and 2 on the replica. Now resume:

docker compose exec replica psql -U postgres -c "SELECT pg_wal_replay_resume();"
sleep 1
docker compose exec replica psql -U postgres -tA -c "SELECT count(*) FROM produtos;"

Expected: 3. For a while, two nodes answered different things and then converged: eventual consistency.

6. Cache-aside with Redis, reading from the replica. Use the cache-aside.sh from the second worked example:

bash cache-aside.sh 1
bash cache-aside.sh 1
bash cache-aside.sh 1
docker compose exec redis redis-cli TTL produto:1

Expected: MISS Camiseta, then HIT Camiseta twice, and an (integer) between 1 and 30 (the seconds of life left on the key).

7. The price of the cache: stale data. The cache key lives only 30 s, so run this step right after step 6. If more time has passed, run bash cache-aside.sh 1 before the UPDATE to repopulate the cache; otherwise the stale data does not appear (you would see MISS Camiseta Azul directly). Change the data and read again:

docker compose exec primary psql -U postgres -c "UPDATE produtos SET nome='Camiseta Azul' WHERE id=1;"
bash cache-aside.sh 1

Expected: HIT Camiseta. The cache serves the old value until it expires. Invalidate when writing:

docker compose exec redis redis-cli DEL produto:1
sleep 1   # gives the replica time to receive the UPDATE (replication is asynchronous)
bash cache-aside.sh 1

Expected: MISS Camiseta Azul. If MISS Camiseta appears, the read reached the replica before the change: this is the replication lag of eventual consistency. Repeat the DEL and the read. Whoever writes needs to invalidate the cache (or accept the TTL as the staleness window).

Exercises

Do the exercises in order: 2 and 3 change the lab's state.

1. Sharding by hash, and the cost of re-sharding. Distribute 10 IDs into 2 shards by modulo (a very simple hash) and then see how many IDs change shard when moving to 3 shards:

   for id in $(seq 1 10); do echo "usuario $id -> shard $((id % 2))"; done
    
   mudou=0
   for id in $(seq 1 1000); do
     # compares the old shard (2 shards) with the new one (3 shards)
     if [ $((id % 2)) -ne $((id % 3)) ]; then mudou=$((mudou + 1)); fi
   done
   echo "IDs que trocaram de shard: $mudou de 1000"

   Expected: IDs que trocaram de shard: 667 de 1000. About two thirds of the data would need to migrate: this is the pain of re-sharding. Databases like Cassandra and DynamoDB do this work transparently. Question: and range sharding, what problem does it have? *Expected answer:* hotspots. As each shard receives a fixed range of values, if the data or the accesses are not evenly distributed across the ranges, one shard gets overloaded while the others sit idle. Two examples: in the A to M and N to Z split, names concentrated in certain letters weigh more on one side; in a split by range of increasing ID (or date), all new writes land on the last shard. In return, range makes range queries easy, which is precisely what hash makes hard.
2. Network partition (CAP in practice). Disconnect the replica from the network, write to the primary and read from both sides:

   docker network disconnect lab-etapa-05_default $(docker compose ps -q replica)
   docker compose exec primary psql -U postgres -c "INSERT INTO produtos VALUES (4,'Chaveiro');"
   docker compose exec primary psql -U postgres -tA -c "SELECT count(*) FROM produtos;"
   docker compose exec replica psql -U postgres -tA -c "SELECT count(*) FROM produtos;"

   Expected: the primary with one more row than the replica. The replica answered (availability) with outdated data (no consistency). Reconnect with docker network connect lab-etapa-05_default $(docker compose ps -q replica) and wait up to about 1 minute: the old replication connection may hang until wal_receiver_timeout expires (60 s by default), and the replica may come back with another IP. To follow along, run docker compose exec replica psql -U postgres -c "SELECT status FROM pg_stat_wal_receiver;"; when replication returns, the status is streaming. Run the count(*) again and check that the numbers converge. Question: if your priority were consistency, what should the replica have done during the partition? *Expected answer:* choosing consistency (CP), the replica should refuse the read, returning an error or leaving rotation, while it could not confirm it is up to date with the primary; the price is availability during the partition. The lab showed the other side, the choice for availability (AP): the replica answered with what it had, and the nodes converged afterwards (eventual consistency).
3. Manual failover and the RTO. Take down the primary and promote the replica:

   docker compose stop primary
   time docker compose exec replica psql -U postgres -c "SELECT pg_promote();"
   docker compose exec replica psql -U postgres -c "SELECT pg_is_in_recovery();"
   docker compose exec replica psql -U postgres -c "INSERT INTO produtos VALUES (5,'Adesivo');"

   Expected: pg_promote returns t, pg_is_in_recovery becomes f and the INSERT works. You have just done a failover. Questions: which part of the RTO was automatic, and which depended on you? With asynchronous replication, what may have been left behind at the moment of the crash (what RPO did you accept)? To restart the lab afterwards, run docker compose down -v and bring everything up again. *Expected answer:* almost none of the RTO was automatic: noticing the outage, deciding to promote and running pg_promote depended on you (a managed service like RDS automates detection and promotion). As for the RPO, with asynchronous replication the transactions already confirmed on the primary that had not yet reached the replica at the moment of the crash are lost; on a local network that is usually little or nothing, but there is no guarantee, and that window is what the RPO measures.
4. RPO and RTO. The business defines an RPO of 5 minutes and an RTO of 15. Synchronous or asynchronous replication? Can the failover be manual? Justify. *Expected answer:* asynchronous replication is enough, because the business accepts losing up to 5 minutes of transactions, as long as the replica lag is monitored and stays within that limit; synchronous would only be needed if the RPO were close to zero. The failover can be manual if the team can detect, decide and promote comfortably within the 15 minutes (with on-call staff and a tested runbook); automating it reduces the risk of blowing the RTO and of human error, at the cost of more complexity.

Cleanup (when you finish the lab and the exercises):

docker compose down -v
cd ..

Lab

1. Bring up and verify.

docker compose up -d --build
sleep 3
bash verificar.sh

Expected:

OK     asset servido pela borda (HIT)
OK     requisições distribuídas em 3 réplicas
OK     carrinho visível em qualquer réplica
OK     pedido aceito de forma assíncrona (202)

2. Resilience: take down a replica.

docker stop $(docker compose ps -q app | head -n 1)
for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code}\n" "localhost:8080/"; done | sort | uniq -c

Expected: 20 200. Bring it back and refresh Nginx, which resolves names only at startup (the replica may come back with another IP):

docker compose up -d
docker compose exec lb nginx -s reload

3. Before and after, in numbers. Use the load function from Stage 01 (redefine it if you opened a new terminal) against the architecture and against a single-instance "monolith":

carga() { time (seq 1 100 | xargs -P 20 -I{} curl -s -o /dev/null "$1"); }
 
# After: edge, LB and 3 replicas
carga http://localhost:8080/
 
# Before: a single stateful instance, with no LB and no edge
docker run -d --name monolito -p 8090:8080 -e MODO=memoria lab-app
carga http://localhost:8090/

Expected: about 1.7 s against about 5 s. Now the single point of failure:

docker stop monolito
curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 localhost:8090/

Expected: 000. Remove with docker rm monolito.

4. Observability: the three pillars with what we have.

*Metrics* (the what):

docker stats --no-stream

Expected: a table with CPU and memory for each container (NAME, CPU %, MEM USAGE / LIMIT...).

*Logs* (the why). The application records each request:

curl -s "localhost:8080/carrinho?user=ana" > /dev/null
docker compose logs --no-color app | tail -n 3

Expected: lines like app-2 | [9f8e7d6c5b4a] GET /carrinho?user=ana rid=4c1d....

*Traces* (the where). Follow a request ID through the layers:

RID=$(docker compose logs --no-color lb | grep carrinho | tail -n 1 | grep -o 'rid=[a-f0-9]*')
[ -n "$RID" ] || echo "nenhum rid encontrado: rode antes o curl do carrinho do passo anterior"
echo "seguindo $RID"
docker compose logs --no-color lb app | grep "$RID"

Expected: two lines with the same rid, one from the lb (with upstream= showing which replica was chosen) and another from the app (the replica that served). It is a mini distributed trace: the same identifier stitching the request's path through two layers. Real tracing tools do this automatically and measure the time of each segment.

Exercises

1. Diagnosis. The e-commerce below went down on Black Friday. List the points of failure and, for each, the piece of the guide that solves it and in which stage it was studied: - application and database on the same server; - cart in a session variable; - images and JavaScript served by the application itself; - confirmation e-mail sent before responding to the customer; - a single server in front of everything. *Expected answer:* application and database on the same server: separate the two, with the application on its own instances and the database on a managed service with read replicas (Stages 01, 03 and 05); cart in a session variable: stateless application, with the state in Redis or in the database (Stage 03); images and JavaScript served by the application: CDN in front (Stage 04); e-mail before responding: queue and workers, with the producer responding right away (Stage 03); single server in front: load balancer with a fleet of instances (Stages 01 and 02).
2. Monolith or microservices? A startup with 6 people, a single team, wants to "be like Nubank". Use Conway's law and the cost table to argue in favor of a scalable monolith (stateless, in an ASG) as a starting point. *Expected answer:* by Conway's law, a single, centralized team tends to produce (and to operate well) a monolith; microservices reflect small, autonomous teams, which the startup does not have yet. The table shows the price of microservices: massive operational complexity, latency and new points of failure in the network, hard distributed debugging. A stateless monolith in an ASG scales horizontally with simplicity of development, testing, deploy and debugging, and can be split later, when teams and bottlenecks arise that justify it.
3. Bottleneck or pillar? You see the latency of /carrinho triple. Which pillar do you consult first to know *that* something changed, which to know *why* and which to know *where* the time was spent? *Expected answer:* metrics to know that something changed (and trigger the alert), logs to know why and traces to know where the time was spent, across the services the request passed through.
4. Error budget (calculation). An SLO of 99.9% availability over 30 days leaves how many minutes of unavailability in the budget? (Hint: 30 days have 43,200 minutes.) *Expected answer:* 0.1% of 43,200 = 43.2 minutes.
5. Upgrade plan. Write what you would add to the docker-compose.yml to include the Stage 05 primary and replica databases behind the API, and how the application would do the write splitting. *Expected answer:* add to the compose the primary and replica services (with the init-primary.sh, the replica.sh and the Stage 05 healthcheck) and pass the app the addresses of both through environment variables. In write splitting, write commands (INSERT, UPDATE, DELETE) go to the primary and reads (SELECT) to the replica, preferably with the Redis cache-aside in front. The test application does not talk to Postgres, so a Postgres client would have to be added to it.

Cleanup:

docker compose down -v
docker rm -f monolito 2>/dev/null
cd ..