Handle Yellowstone gRPC reconnects and stream load

Plan Yellowstone gRPC reconnects, gap recovery, bounded queues, message limits, and monitoring for a long-running consumer.

A long-running consumer needs a recovery plan as well as a fast connection. Measure incoming rate, processing rate, and lag before changing concurrency or server settings.

Keep the working set small

Filter by address or program before decoding. Prefer block metadata when you only need hashes and counts. Slice account data only when the decoder expects that slice. Group quiet subscriptions on one connection; split high-volume programs after measurements show contention.

Bound work and memory

Awaiting an async handler controls concurrency, but it does not make CPU-heavy work parallel. Use a bounded queue and workers for expensive parsing. Define what happens when the queue fills: pause, sample disposable metrics, or stop and recover durable data. Track queue depth and the oldest pending slot.

Reconnect with a retry budget

Save the quickstart helper as stream.mts in the same directory. These examples use the same environment variables and dependencies. This example retries the metadata stream with backoff and jitter, then exits after eight interruptions. Authentication failures should be corrected before restarting.

reconnect.mts

import { setTimeout as delay } from "node:timers/promises";
import { subscribe } from "./stream.mts";

let retry = 0;
while (retry < 8) {
  try {
    for await (const update of subscribe({ blocksMeta: { blocks: {} } })) {
      if (update.blockMeta) console.log(update.blockMeta.slot);
    }
    console.error("Stream ended");
  } catch (error) {
    console.error("Stream failed", error);
  }
  retry += 1;
  const waitMs = Math.min(30_000, 500 * 2 ** retry) + Math.random() * 500;
  await delay(waitMs);
}
throw new Error("Reconnect budget exhausted");

Recover gaps explicitly

Persist progress only after the downstream write succeeds. Reconnect with the full subscription and reconcile the missing range using available RPC history or a fresh account snapshot. Use replay only when your endpoint confirms support and retention; do not assume every stream can replay arbitrary slots. Deduplicate replayed updates.

Tune from measurements

• Compare EU and US delivery from the deployment host. ICMP ping alone does not measure stream latency.
• If a message exceeds the client limit, first reduce the payload. Configure a larger receive limit using the options supported by your client version and budget for its memory cost.
• Respond to server pings. Transport keepalive and subscription ping messages are different mechanisms.
• Alert on repeated disconnects, decode errors, growing queues, stalled progress, and write failures.
• Test interruption and recovery with your actual downstream database before relying on the consumer.

Related guides

Authentication

Quickstart