Crypto bots after Web3: rate limits, kill switches, honest backtests

When the REST call that places an order times out, the bot does not know whether the exchange accepted the order. Retry blindly and it may create a duplicate. Questions like this one came up when, after the smart contract and token presale projects, we turned to exchange automation. First, a disclaimer: this article is not investment advice. We build the infrastructure for trading and automation bots; we do not build trading strategies for clients’ money. The strategy always belongs to the client. Our job is to make the system do exactly, predictably and safely what the strategy prescribes.

Two channels, two truths

Nearly every centralised exchange offers two channels. Over the REST API we place and cancel orders and query balances and history. Over WebSocket come prices, order book updates and status changes of our own orders. The two are not in sync. WebSocket may already show a fill while REST still reports the order as “open”, and the reverse happens too.

Our answer: WebSocket is the primary source of state, a periodic REST reconciliation runs behind it, and when the two disagree, the exchange’s REST response wins. A heartbeat watches the WebSocket connection. On reconnect we request a full snapshot, because backfilling missed messages works differently on every exchange and is rarely reliable.

The order that timed out

An order’s state machine looks simple: created, open, partially filled, filled, cancelled, rejected. The difficulty lies in the transitions, and the timeout is the worst of them. So every order gets its own client-side identifier, which most exchanges accept. A repeat then creates no duplicate; the exchange returns the order that already exists.

Every state change goes into an append-only event log in PostgreSQL, and current state is derived from it. If anything is disputed, the event sequence shows what happened and when.

Rate limits

The other wall every bot hits is the rate limit. Exchanges weight their limits: fetching an order book costs more than a ping. Going over brings a short ban, which hurts precisely when an order needs cancelling fast. So:

• a client-side token bucket tracks consumption, read back from the exchange’s own headers;
• critical operations (cancellation, kill switch) get reserved headroom that queries cannot use up;
• errors trigger exponential backoff with jitter, and after a 429 response we wait as long as the exchange asks, not as long as we would choose.

The layer beneath the strategy

For us, risk management is independent of the strategy. It runs underneath, and the strategy cannot bypass it:

• position limits per asset and in aggregate; above the limit the system places no opening orders;
• a maximum daily loss, after which the bot permits only closing orders;
• a kill switch: one command that cancels every open order and puts the bot into passive mode, reachable over a separate channel as well, independent of the bot’s own web interface;
• a paper trading mode in which everything runs the same way, except that orders go to a simulated executor.

Paper trading is the foundation of testing, and we do not treat it as a convenience. Every code change runs there first, for days, on the live data stream.

Failing silently

A bot that stops silently is worse than one that fails loudly. Its state is therefore reported by heartbeat metrics: the age of the last price update, the number of open orders, the connection state. An alert fires if any of them leaves the expected range, or if the data stream stalls even without an error message. Alerts go out over several channels, because the bot itself cannot know which one is alive at the time.

API keys never end up in the repository or in the Docker image. They come from a secret store at startup, as environment variables. Their permissions are minimal (trading yes, withdrawal no) and they are IP-restricted. Revocability matters more than encryption: a compromised key has to be replaceable within minutes.

Pretty curves, bad assumptions

Backtesting is seductive because it draws pretty curves, but the result is only as good as its assumptions. The typical errors we can deal with in software:

• look-ahead: the strategy uses data that was not yet available at that moment, so the backtesting engine serves only data that had “already arrived” according to its timestamp;
• slippage and fees: the simulated executor fills based on the order book, with fees, never at the mid price;
• missing data: we do not interpolate missing candles; we mark them, and the strategy sees the gap too;
• non-deterministic runs: the same input always produces the same result, because randomness is seeded and time is injected instead of being read from the system clock.

This layer guarantees that a backtest is reproducible and honest. It does not guarantee that the strategy is good. The two have to be kept apart, and in development we take responsibility for the first.