Load-balance Node.js containers with nginx and Docker Compose, and scale without reloading nginx

One Node.js process runs your JavaScript on one CPU core. For I/O-heavy apps that's rarely the bottleneck, but anything CPU-bound in a request handler (templating, image work, hashing, big JSON) caps a single process at one core's worth of throughput. A simple way to use the whole machine is to run several identical containers and put nginx in front of them.

This post builds that with Docker Compose, measures whether it helps, and covers the part most tutorials skip: scaling up and down while traffic is flowing. Everything was run with Docker 29 and Compose v2 on a 4-CPU Docker Desktop VM, using node:24-slim and nginx:1.31 (1.31.6).

The app

app/server.mjs answers with the container's hostname, so we can see which replica handled a request. /work burns about 50 ms of CPU to simulate a CPU-bound handler. It also shuts down gracefully, which matters when replicas are removed. The graceful-shutdown details are explained in Running Node.js in Docker.

import { createServer } from 'node:http';
import { pbkdf2Sync } from 'node:crypto';

let shuttingDown = false;

const server = createServer((req, res) => {
  // /work burns ~CPU so we can see whether more replicas help
  if (req.url === '/work') pbkdf2Sync('secret', 'salt', 50_000, 64, 'sha512');
  if (shuttingDown) res.setHeader('connection', 'close');
  res.end(`Hello from ${process.env.HOSTNAME}\n`);
});
server.listen(3000, () => console.log(`listening on :3000 as ${process.env.HOSTNAME}`));

function shutdown() {
  shuttingDown = true;
  server.close(() => process.exit(0));
  setTimeout(() => process.exit(1), 8000).unref();
}
process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

FROM node:24-slim
WORKDIR /app
COPY app/ ./
USER node
CMD ["node", "server.mjs"]

Compose and nginx

compose.yaml. The top-level version: key is obsolete in Compose v2, so leave it out:

services:
  app:
    build: .
    init: true
    deploy:
      replicas: 4

  nginx:
    image: nginx:1.31
    ports:
      - "8080:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - app

nginx.conf:

events {}

http {
    # Docker's embedded DNS server. valid=10s: re-check the replica list every 10 seconds.
    resolver 127.0.0.11 valid=10s ipv6=off;

    upstream app {
        zone app 64k;          # shared memory, required for "resolve"
        least_conn;            # send each request to the replica with the fewest active connections
        server app:3000 resolve;  # one entry per IP that "app" resolves to, kept up to date
        keepalive 16;          # reuse connections to the Node processes
    }

    server {
        listen 80;

        location / {
            proxy_pass http://app;
            proxy_http_version 1.1;          # needed for upstream keepalive
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_connect_timeout 1s;           # a removed replica's IP doesn't refuse, it just hangs
            proxy_next_upstream error timeout;  # …so give up quickly and retry another replica
        }
    }
}

The key line is server app:3000 resolve;. Compose gives every replica of the app service the DNS name app on the project network, and Docker's embedded DNS server (127.0.0.11) returns one address per running replica. With resolve, nginx turns each address into an upstream server, re-checks the name every valid=10s, and updates the list without a reload. This used to be an NGINX Plus feature. Since nginx 1.27.3 it's in open-source nginx, and it needs the shared-memory zone line.

$ docker compose -p balancer up -d --build
$ docker compose -p balancer ps --format '{{.Name}}  {{.Status}}'
balancer-app-1  Up 2 seconds
balancer-app-2  Up 3 seconds
balancer-app-3  Up 2 seconds
balancer-app-4  Up 2 seconds
balancer-nginx-1  Up 2 seconds

Compose v2 names containers with hyphens (balancer-app-1), not the underscores (balancer_app_1) of the old docker-compose v1. That breaks scripts that hard-code container names.

Is the load actually balanced?

400 sequential requests, counted by the hostname in the response:

nginx.conf (resolve + least_conn)    requests per replica: 100 100 100 100

Exactly even. For comparison, the configuration from the first version of this post (and many other tutorials) gave this:

old config                           requests per replica: 124 104 94 78

That config defined an upstream app { least_conn; server 0.0.0.1; } block with a placeholder address, then proxied to a variable: set $upstream app; proxy_pass http://$upstream:3000;. It "works", but not the way it reads. nginx's docs say that with a variable, the name "is searched among the described server groups, and, if not found, is determined using a resolver". A group without a port doesn't match a proxy_pass that includes one. So the :3000 made nginx skip the upstream block entirely, resolve app through Docker's DNS, and pick among the addresses on its own. least_conn never applied. Two tests confirm it:

• Deleting the whole upstream block made no difference (106 / 100 / 98 / 96).
• Removing :3000 from proxy_pass makes nginx use the group after all, and every request then goes to the placeholder: 502, connect() failed (111: Connection refused) … upstream: "http://0.0.0.1…".

Does it help? 1 vs 2 vs 4 replicas

autocannon ran inside the Compose network for 15 seconds with 16 concurrent connections, hitting the CPU-bound /work endpoint through nginx:

Throughput scales almost linearly with the number of replicas, up to the number of CPUs. It's a little under 4× at four replicas, because nginx and the load generator use the same four cores. Going beyond your core count won't help a CPU-bound workload. For an I/O-bound app (waiting on a database or another API), one process often handles a lot already, so measure before you multiply.

Scaling up without touching nginx

$ docker compose -p balancer up -d --scale app=6 --no-recreate
2s after scale-up: 4 replicas answering
7s after scale-up: 6 replicas answering

No reload and no restart: the new replicas started receiving traffic within the 10-second valid window. Lower valid if you need nginx to notice changes faster.

Scaling down without dropping requests

This is the part that went wrong at first. Scaling from 6 to 2 replicas while sending 600 requests, without the proxy_connect_timeout 1s line:

during scale-down 6→2:  4 × no response (curl gave up after 3s), 596 × 200

nginx still had the removed replicas' IP addresses in its list until the next DNS check. A container that is stopping refuses connections, and nginx retries another replica right away. A container that is already gone leaves an IP that nobody answers, so the connection attempt just hangs until proxy_connect_timeout runs out. That defaults to 60 seconds. With the 1-second timeout and proxy_next_upstream error timeout, two scale-down rounds of 600 requests each:

round 1, scale-down 6→2:  600 × 200 | slowest: 2.0s
round 2, scale-down 6→2:  600 × 200 | slowest: 3.0s

nginx error log:
   1 connect() failed (111: Connection refused) while connecting to upstream
   7 upstream timed out (110: Connection timed out) while connecting to upstream

Every request succeeded. The log shows exactly the two cases: one refused connection (a replica shutting down) and seven connect timeouts (IPs that had already disappeared), each retried on a live replica. A 1-second connect timeout is safe on a local Docker network, where a healthy replica accepts a connection almost instantly. Across a real network, use a few seconds.

Note that proxy_next_upstream only retries while nothing has been sent to the client yet. It also won't resend a non-idempotent request like POST once it has reached a replica, unless you add non_idempotent, and that's the right default. A POST that failed to connect, as above, is still retried.

When to use something else

• One machine, no Docker: node:cluster or a process manager can run one worker per core behind a single port.
• CPU-heavy work inside requests: moving it to worker_threads keeps the event loop responsive, even with one process.
• Several machines: this is what Kubernetes Services, ECS or a cloud load balancer do for you, including health checks and draining connections from replicas that are being removed.

Sources & further reading

• nginx: upstream server parameters (resolve, open source since 1.27.3) and least_conn
• nginx: proxy_pass (how names in variables are resolved), proxy_connect_timeout and proxy_next_upstream
• nginx: using nginx as an HTTP load balancer
• Compose file reference: deploy.replicas and the obsolete version key
• Docker networking: embedded DNS
• autocannon