Hundreds of requests a second: what our first high-load APIs taught us

Hundreds of requests per second, with response time feeding straight into the client’s business. Over the past six months we have handed over several backends like that, and the same problems kept coming back. None of the fixes is revolutionary. What follows is closer to a checklist we wish we had written for ourselves sooner.

The database runs out first

The Node.js event loop almost never runs out. Under NestJS, a well-written endpoint handles thousands of requests per second in a single process, as long as it is not waiting on the database. In practice it is nearly always waiting on the database.

So in every project we do three things:

• Indexes based on real queries. In production, step one is switching on pg_stat_statements. We go through the slowest and most frequent queries with EXPLAIN (ANALYZE, BUFFERS) and only index the column combinations where the plan actually shows a sequential scan. “Index every foreign key” is a decent starting point. It is no substitute for measuring.
• Deliberate connection pooling. The number of Node.js processes multiplied by the pool size must stay below PostgreSQL’s max_connections, with headroom left for maintenance. With multiple instances, PgBouncer sits between the processes and the database in transaction mode. That means giving up session-level settings and prepared statements, which is better known in advance.
• Short transactions. We never call an external service inside a transaction. When it cannot be avoided, the transaction is closed before the call and the result written back separately.

Redis, three jobs

Redis is our cache, our rate limiter and our lock, and each role needs its own configuration.

Cache. For read-heavy endpoints (lists, configuration, rarely changing master data) we store the response by key with a short TTL. We do not attempt perfect cache invalidation. A few seconds of staleness is usually acceptable, and far simpler than event-driven eviction tied to writes. Where it is not acceptable, we do not cache.

Rate limiting. A sliding-window counter per client (by API key or IP), with the X-RateLimit-* headers returned in the response. The check lives in a global NestJS guard, outside the individual endpoints, so nobody can forget it.

Distributed lock. When more than one instance can modify the same resource, such as when processing an external payment callback, we take a short-lived SET NX lock. A single Redis instance, no Redlock. Fault tolerance comes from idempotent processing; the lock plays no part in it.

Idempotency and pagination

Both are easy to neglect and hard to retrofit.

An idempotency key is mandatory on every mutating endpoint that an external system or a mobile app can call. The client sends an Idempotency-Key header, the server keeps the response for that key for a while and returns the same response to a repeat call. That is what protects the system from duplicate orders and messages caused by network retries.

For pagination, OFFSET slows down noticeably beyond tens of thousands of rows, and on a moving data set it skips or repeats items. We moved to cursor-based pagination: the client receives the sort key of the last item it saw and uses it to request the next page. It is less convenient for “jump to page 47” interfaces. For APIs it is nearly always better.

Watch the p99

The main lesson from the first weeks in production: log files alone are not enough. Every system we build gets:

• structured (JSON) logging with a request ID that can be traced from the incoming request down to the database query,
• Prometheus metrics per endpoint: request count, error rate, response-time percentiles (p50, p95, p99),
• the state of the database pool (waiters, busy connections) as a metric,
• a health check endpoint that verifies both the database and Redis, watched by both Docker and Nginx.

The number we watch is p99 response time. The average nearly always looks good; p99 shows what one percent of users actually go through.

Load testing before launch

We do not trust our own estimates. Before every launch we run a scenario with k6 or a similar tool that simulates a multiple of the expected peak load, against a database filled with a realistic volume of data. The test runs on staging, which runs in Docker with the same configuration as production.

The usual finds: a missing index, a pool that is too small, an N+1 query hidden by the ORM, or an endpoint that returns several megabytes of JSON for a single request. Each one is cheaper to catch in a staging test than on the first day of a campaign.

The next post covers multi-tenant SaaS architectures, where these problems gain another dimension: tenants affecting each other.

We also build high-load backends as part of SaaS projects: SaaS development and operations.