Yellowstone gRPC Reconnect and Replay: A Production Guide

A Yellowstone subscription can stay healthy for hours, fail for seconds, and leave a trading bot with a local state that looks current but is not. The socket came back. One pool update did not. That is enough to make the next decision against the wrong reserves.

This guide defines the recovery contract around the stream. It complements the Yellowstone filter cookbook, which covers what to request. Here the question is what happens after the request stops receiving data.

Separate connection liveness from data continuity

Track these as different states:

An end, close, or recoverable transport error should move the consumer out of healthy. So should a stale-data timer, even when the channel still reports connected.

Use an explicit state machine:

STARTING -> LIVE -> STALE -> RECONNECTING -> REPLAYING -> RECONCILING -> LIVE
                         \-> GAP_TOO_OLD -> PAUSED -> REBUILDING ---^

Do not let the execution worker read a model in STALE, REPLAYING, GAP_TOO_OLD, or REBUILDING. Recovery is part of the risk gate, not a background log message.

Checkpoint complete slots, not the last object received

A transport can fail halfway through a slot. Saving the slot number from the last account update can therefore overstate what the process has fully consumed.

The built-in Yellowstone reconnect design checkpoints completed slots from block metadata. On a recoverable error it requests a few slots before that checkpoint, then removes duplicates from the overlap. Triton documents this behavior for the Rust client added in v13.1. See the auto-reconnect release note and the Yellowstone repository.

The current Rust client setup is small:

use yellowstone_grpc_client::{
    GeyserGrpcClient,
    ReconnectConfig,
};
 
let mut client = GeyserGrpcClient::build_from_shared(endpoint)?
    .x_token(Some(token))?
    .set_reconnect_config(ReconnectConfig::default())
    .connect()
    .await?;
 
let mut stream = client.subscribe_once(request).await?;

Treat this as a client capability, not a universal endpoint guarantee. The endpoint must run a compatible server and retain the requested history. Ask for the first available replay slot before you depend on backfill.

Use overlap, then deduplicate before side effects

Exact resume boundaries are brittle. A short overlap is safer because it covers a failure that arrived between the last update and the durable checkpoint write.

Overlap creates duplicate delivery by design. Build the consumer so the same update converges:

Deduplication in memory protects one process lifetime. Trading side effects need durable idempotency. Write the observed transaction signature as a unique key before a worker submits a copy, rebalance, alert, or webhook.

type SeenTransaction = {
  signature: string;
  observedSlot: bigint;
  actionSignature: string | null;
};
 
async function claimCandidate(signature: string, slot: bigint): Promise<boolean> {
  const inserted = await store.insertIfAbsent<SeenTransaction>({
    signature,
    observedSlot: slot,
    actionSignature: null,
  });
 
  return inserted;
}

insertIfAbsent must be backed by a unique constraint or an equivalent atomic operation. A check-then-insert sequence can race when two workers receive the same replayed event.

Know the replay boundary before the incident

The Yellowstone protobuf includes from_slot and a SubscribeReplayInfo method that reports the first available replay slot. Those fields are part of the published protocol.

The provider decides how much history is retained. Triton's May 2026 note states that Dragon's Mouth currently keeps about 1,000 slots. That number describes that service at that date, not every Yellowstone endpoint and not a permanent contract.

At reconnect time, compare three values:

requested_from = last_complete_slot - overlap_slots
first_available = endpoint replay boundary
 
if requested_from < first_available:
  recovery = GAP_TOO_OLD
else:
  recovery = REPLAYING

When the gap is too old, do not jump to the live head and continue silently. Record the missing range, stop dependent execution, reload the accounts or application state through JSON-RPC, and enable the strategy only after reconciliation passes.

Keep the channel alive without hiding a stale stream

The official Yellowstone repository notes that some load balancers close bidirectional streams when the client sends nothing for a period. The protocol includes a ping request for this case. The server can send a ping and the client can answer without replacing its filters.

A keepalive policy needs two timers:

1. Transport timer: respond to the protocol ping and reconnect on channel failure.
2. Data timer: alert when the last relevant slot or block update is older than the workload's allowed staleness.

The second timer catches a connection that is technically open but operationally useless. Choose its threshold from observed slot cadence and the workload, then test it during a controlled network pause.

Reconcile after every uncertain interval

Replay restores events. It does not establish that your reducer, decoder, queue, and side effects all reached the same state as the chain.

After reconnect:

• fetch the authoritative accounts that drive the next decision;
• compare their slot and decoded values with the local model;
• verify pending transaction signatures independently;
• clear or rebuild derived caches if the comparison fails;
• return to LIVE only after the check passes.

For a copy bot, reconcile the watched and mirror signatures. For a DLMM worker, reload the pair, active bin, relevant bin arrays, and inventory accounts. The recovery procedure should name the state it verifies rather than run one generic getSlot call.

Metrics that make a stream operable

Log recovery as a bounded event with these fields:

Test the path by cutting the connection for a short interval, then for longer than the documented replay window in a non-production environment. Both cases need a deterministic result.

The production rule

Recovery is complete when the consumer can account for the interval, remove duplicates, reconcile state, and re-enable actions deliberately. A green connection icon covers only the first step.