Zero-Downtime Solana RPC Migration Runbook

Changing a Solana endpoint can take seconds. Proving that the new path preserves application behavior takes longer.

An application may use the same provider for account reads, blockhashes, transaction simulation, transaction delivery, WebSocket subscriptions, and Yellowstone gRPC. Those paths do not fail in the same way. A clean migration separates them and preserves a working return route at every stage.

This runbook uses eight states:

inventory -> qualify -> shadow -> compare -> canary -> cut over -> observe -> retire
                 |          |          |           |
                 +----------+----------+-----------+-> rollback

Inventory the application contract

Start with what the application sends, not a provider feature table.

List every RPC method, commitment level, timeout, retry rule, region, expected response size, and downstream owner. Add WebSocket subscriptions and gRPC filters. On transaction paths, document where the blockhash comes from, where simulation runs, which endpoint submits the signed bytes, and how confirmation is checked.

The inventory should answer these questions:

• Reads that block a user action or trading decision.
• Cached responses and their retention time.
• Requests that rely on minContextSlot.
• Transaction errors that are retried, surfaced, or ignored.
• Events that can arrive through both polling and streaming.
• The durable key that prevents replay from repeating a business action.
• The operator who can return each traffic class to the old provider.

Method support is binary. Operational behavior is not. Two providers may both support getProgramAccounts while applying different limits, timeouts, or response-size controls to the application's workload.

Qualify identity before performance

The replacement endpoint must point at the intended cluster. The getGenesisHash method returns the connected cluster's genesis hash. Verify it before sending shadow traffic. Record getVersion as environment evidence, not as a quality score.

Then check liveness and freshness separately. Solana defines getHealth as healthy when the node is within its configured slot distance of the cluster tip. That makes it a useful node signal, but not an application-level acceptance test.

Sample getSlot at the same commitment from the old endpoint, the candidate, and an independent reference. Keep the timestamp and region with every sample. The method returns the highest slot reached at the requested commitment, so comparisons only make sense when commitment and sampling time match.

type RpcResult<T> = { jsonrpc: "2.0"; id: number; result: T };
 
async function rpc<T>(url: string, method: string, params: unknown[] = []): Promise<T> {
  const response = await fetch(url, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ jsonrpc: "2.0", id: 1, method, params }),
  });
 
  if (!response.ok) throw new Error(`${method} returned HTTP ${response.status}`);
  const body = (await response.json()) as RpcResult<T>;
  return body.result;
}
 
export async function qualify(url: string) {
  const [genesisHash, health, finalizedSlot, version] = await Promise.all([
    rpc<string>(url, "getGenesisHash"),
    rpc<string>(url, "getHealth"),
    rpc<number>(url, "getSlot", [{ commitment: "finalized" }]),
    rpc<Record<string, string>>(url, "getVersion"),
  ]);
 
  return { sampledAt: new Date().toISOString(), genesisHash, health, finalizedSlot, version };
}

Do not copy a universal acceptable slot gap from another team. Set the limit from the application's current baseline, commitment choice, region, and consequence of stale data.

Shadow reads and compare semantics

Shadowing sends a copy of real read traffic to the candidate while the old provider remains authoritative. The candidate response is measured and discarded.

Capture enough context to explain a mismatch:

Do not demand byte-for-byte equality at the live tip. Nodes can answer from different slots. Define method-specific invariants instead:

• Cluster identity must match exactly.
• Schema and required fields must match the application's parser.
• Account owner and executable status should agree at a comparable slot.
• Confirmed or finalized state should converge within the team's accepted window.
• Errors must be classified consistently enough for the retry policy.
• Large production-shaped requests must complete inside their allocated deadline.

Sample both ordinary and difficult traffic. A benchmark made only from getBalance says little about a workload dominated by large account scans or transaction history.

Treat blockhashes and transaction delivery as a separate cutover

The getLatestBlockhash method returns a recent blockhash with its last valid block height. Store both with the transaction attempt. Use isBlockhashValid when diagnosing whether a candidate considers that blockhash valid at the selected commitment.

Before submitting real canaries, run representative payloads through simulateTransaction. Simulation is a compatibility check, not proof that a later send will land.

With real traffic, keep one authoritative submission decision per signed transaction. The sendTransaction documentation notes that a successful RPC response means the node accepted the request, not that the transaction was confirmed. Record the signature, endpoint, region, blockhash, last valid block height, submission time, RPC result, and independent confirmation result.

Move the send path in small cohorts. Measure landing and error outcomes for the same transaction class. Never spray a transaction to multiple providers unless the application has explicitly designed for duplicate submission and can still attribute the outcome.

Dual-run Yellowstone streams before moving consumers

A healthy socket can carry stale, incomplete, or misfiltered data. Run the old and candidate streams at the same time and give each update this envelope:

type StreamObservation = {
  source: "old" | "candidate";
  generation: number;
  slot: bigint;
  receivedAtMs: number;
  kind: "transaction" | "account" | "slot";
  identity: string;
};

The transaction signature is its durable base identity. An account update identity should include the account address, slot, and write version where the service exposes it. Treat source and reconnect generation as observation metadata, not in the business-event key.

Measure coverage, first-arrival time, duplicate rate, quiet periods, reconnect behavior, and queue age. The upstream Yellowstone gRPC repository documents subscription filters and replay from a slot. Provider limits and retention can differ, so validate the exact production filter against the service you plan to use.

If both streams feed one consumer during the test, deduplicate before side effects. A second observation may update status or telemetry. It must not repeat an alert, order, webhook, or ledger entry.

Canary in an order you can reverse

Use independent routing controls for each traffic class. A practical order is:

1. Internal and non-critical reads.
2. A bounded share of user-facing reads.
3. One read-only stream consumer.
4. Remaining stream consumers by shard.
5. Transaction simulation.
6. A bounded cohort of transaction sends.
7. Full traffic after the observation window passes.

At every step, record the previous routing value and the command that restores it. Leave the old credentials, quotas, connections, and dashboards live through the rollback window.

Pause the migration when any stop condition fires. Useful stop conditions cover stale slots, unexpected error classes, missing stream updates, growing consumer queue age, confirmation regression, or a material cost anomaly. The numerical threshold belongs in the team's acceptance sheet, not in a generic article.

Use the RPC Migration Acceptance Sheet

Complete this before cutover. An empty cell is a failed gate.

Retain the sheet with the deployment record. It turns "the new endpoint looked fine" into evidence that another operator can audit.

Retire the old path last

Cutover is not completion. Continue comparison during a defined observation window. Reconcile transactions and durable stream state, then confirm that retries, backfills, and scheduled jobs no longer reference the old endpoint.

Only then remove credentials and capacity. Retain the migration record, baselines, and rollback notes for the next provider or regional failover exercise.

If you want to qualify RPCEdge with your own method mix, transaction path, and Yellowstone filters, start with the RPCEdge plans and endpoints, then run this acceptance sheet against both paths.