Observability for a Solana Trading Stack

Measure the pipeline, not the endpoint

A green RPC health check can coexist with a losing trading system. The endpoint may answer while your stream lags, decoder queues, strategy misses its deadline, sender targets a stale leader, or transaction expires unseen.

The unit of observability should be the full action path:

source → ingest → decode → state lookup → decision → sign → submit
                                                    ↓
finalized ← confirmed ← processed ← first-seen ← delivery path

Each arrow can add delay or lose information. One end-to-end number tells you that the outcome changed. It does not tell you why.

Building an HFT Data Pipeline on Solana defines the system stages. This guide turns those stages into traces, metrics, logs, SLOs, and alerts.

Start with one correlation model

Three identities matter:

One intent may produce several signatures if a blockhash expires and the transaction is rebuilt. One signature should never represent two intents.

That distinction prevents a common reporting error. Counting signatures as independent trades inflates attempt volume, while collapsing every retry into one row can hide duplicate execution risk.

Use a stable internal intent_id across rebuilds. Attach signature only after signing.

The event record

Store append-only events rather than updating one mutable status row. A minimal schema looks like this:

type TradingTelemetryEvent = {
  observedAt: string;          // wall-clock UTC for cross-system joins
  monotonicNs?: bigint;        // local duration math
  event: string;               // e.g. "tx.first_seen"
 
  opportunityId?: string;
  intentId?: string;
  signature?: string;
  traceId?: string;
 
  strategy: string;
  transactionClass?: string;
  source?: "shred" | "grpc" | "rpc";
  route?: "tpu" | "jito" | "rpc-forward";
  clientRegion: string;
  serverRegion?: string;
 
  slot?: number;
  leader?: string;
  commitment?: "first-seen" | "processed" | "confirmed" | "finalized";
  outcome?: "landed" | "expired" | "failed" | "skipped";
 
  durationUs?: number;
  computeUnitLimit?: number;
  computeUnitPrice?: number;
  jitoTipLamports?: number;
  errorCode?: string;
};

The type is illustrative, not a protocol standard. Keep names stable in your own system. OpenTelemetry semantic conventions provide common names for standard operations; Solana-specific attributes will remain custom until a shared convention exists.

Timestamp every stage once

Record events at boundaries, not in the middle of an ambiguous function.

t_source_receive
t_decode_complete
t_state_ready
t_decision
t_sign_complete
t_submit_start
t_gateway_ack
t_first_seen
t_processed
t_confirmed
t_finalized

These timestamps produce distinct durations:

Never rename gateway ACK to landing latency. The gateway can accept a transaction that later expires, fails, or never reaches the relevant leader.

Use monotonic clocks for durations

Wall-clock timestamps are necessary when joining events across machines. They can jump when a clock is corrected. Local duration math should use a monotonic clock that only moves forward.

For a span contained inside one process:

const started = process.hrtime.bigint();
 
await decodeBatch(batch);
 
const durationUs = Number(process.hrtime.bigint() - started) / 1_000;

Cross-machine subtraction still depends on clock synchronization. Monitor clock offset and uncertainty as first-class infrastructure metrics. If uncertainty is larger than the latency difference you are trying to claim, the benchmark cannot support the claim.

Traces explain one action

A distributed trace is useful for following one opportunity through multiple processes:

opportunity.evaluate
├─ stream.receive
├─ transaction.decode
├─ state.lookup
├─ strategy.decide
├─ transaction.build
├─ transaction.sign
├─ sender.submit
└─ settlement.reconcile

Propagate trace context through internal HTTP, gRPC, and queue boundaries. The OpenTelemetry context propagation guide explains how trace and parent identifiers correlate work across services.

Avoid sending internal baggage to public RPC endpoints unless required and reviewed. Never place keys, wallet secrets, customer identifiers, or strategy parameters in trace baggage.

An asynchronous on-chain observation is not always a child of the submit request. Link the observation span to the submission trace using the signature or intent record. Forcing every asynchronous relationship into a parent-child tree makes traces misleading.

Metrics describe the distribution

Traces answer "what happened to this action?" Metrics answer "how is this class of actions behaving?"

Create histograms for durations:

solana_stream_delivery_duration_us
solana_decode_duration_us
solana_decision_duration_us
solana_sign_duration_us
solana_submit_ack_duration_us
solana_submit_to_first_seen_duration_us
solana_submit_to_processed_duration_us
solana_submit_to_confirmed_duration_us

Create counters for outcomes:

solana_opportunities_total
solana_transaction_intents_total
solana_transaction_attempts_total
solana_transactions_landed_total
solana_transactions_expired_total
solana_transactions_failed_total
solana_stream_gaps_total
solana_retries_total
solana_duplicate_intents_total

Useful bounded dimensions include:

• Strategy class
• Transaction class
• Read source
• Send route
• Client and server region
• Commitment
• Outcome
• Error class
• Fee percentile bucket
• Leader-distance bucket

Avoid using signature, blockhash, wallet address, account address, slot, trace ID, or error message as a metric label. Those values have high or unbounded cardinality and create a new time series for nearly every event.

OpenTelemetry's metrics guide explains that each unique attribute combination needs aggregation state. It also documents the SDK cardinality limit and overflow behavior. Put high-cardinality identifiers in traces or structured logs instead.

Logs preserve exact evidence

Structured logs are the right place for one-off details:

{
  "event": "tx.first_seen",
  "intent_id": "int_01J...",
  "signature": "5K7...",
  "route": "tpu",
  "slot": 356000001,
  "client_region": "fra",
  "leader_region": "ams",
  "submit_to_first_seen_us": 68421
}

Logs should be machine-queryable and redacted at the source. Never log private keys, seed phrases, raw authorization headers, signed strategy configuration, or customer payloads by default.

Attach trace_id and span_id when available. The OpenTelemetry signals model separates traces, metrics, and logs while allowing them to describe the same underlying activity.

Commitment is a sequence, not one status

Preserve every observation:

first-seen → processed → confirmed → finalized
       ↘ missing / fork-discarded / expired

First-seen is not a Solana RPC commitment. It is a propagation-layer observation. Processed, confirmed, and finalized represent progressively stronger states.

The official signatureSubscribe method can notify a client when a signature reaches the requested commitment. The getSignatureStatuses method supports explicit status checks and reconciliation. Use an independent observation path when measuring a sender so the system under test is not grading its own result.

Solana Commitment Levels Explained covers the safety properties of each level.

Landing rate needs a denominator

"98% landing rate" is meaningless without eligibility rules. Define the denominator before the test.

For example:

eligible attempts:
- signed with a valid recent blockhash
- passed local simulation or was intentionally sent without it
- submitted before the opportunity deadline
- had enough balance for fee and execution
- not canceled by strategy policy
 
landing rate = landed eligible intents / all eligible intents

Report excluded attempts by reason. Otherwise a strategy can improve its displayed landing rate by silently dropping difficult transactions before they enter the denominator.

Track both intent and attempt views:

• Intent landing rate: did the business action land at least once?
• Attempt landing rate: what share of submitted signatures landed?
• Duplicate execution rate: did one intent land more than once?
• Expiry rate: did every signature for an intent age out?

The last two expose retry policies that look successful while creating economic risk.

Fee and route telemetry belong beside landing

A transaction outcome needs its execution context:

compute_unit_limit
simulated_compute_units
compute_unit_price_micro_lamports
estimated_priority_fee_lamports
jito_tip_lamports
send_route
target_leaders
blockhash_age_at_submit
retry_number

Group landing rate and time-to-land by fee bucket and route. A rising fee can correlate with congestion, so an unadjusted chart may misleadingly show high fees performing worse. Segment by congestion regime, transaction class, and writable-account set when the data permits.

Solana Priority Fees Explained provides the fee-policy model, while Landing Transactions on Solana covers direct TPU and Jito paths.

Four dashboards are enough to start

1. Signal health

• Events per second by source
• Source lead rate
• Stream reconnects and gaps
• First arrival to decode p50, p95, p99
• Fork-discarded first-seen observations

2. Strategy health

• Opportunities and skip reasons
• State age at decision
• Decision duration p50, p95, p99
• Expected edge distribution
• Queue depth and dropped work

3. Landing health

• Eligible intents, attempts, and landed intents
• Submit-to-first-seen and submit-to-confirmed distributions
• Expiry, failure, and duplicate execution rate
• Retry count and blockhash age
• Results by route, region, leader distance, and fee bucket

4. Economic outcome

• Expected versus realized edge
• Fees and tips paid
• Slippage
• Failed-action cost
• Profit or loss by latency and landing bucket

The economic view stops infrastructure optimization from becoming a latency contest with no business outcome. A route that saves time but costs more than the edge is not an improvement.

SLOs and alerts should pair speed with correctness

Avoid an alert that fires only on latency. A stream can look fast because it stopped delivering difficult events.

Pair indicators:

stream p99 + stream completeness
decode p99 + decoder error rate
gateway ACK p99 + eligible intent landing rate
submit-to-first-seen p99 + missing-observation rate
confirmed latency + expiry rate
retry count + duplicate execution rate

Example SLO shapes:

• 99.9% of expected stream sequence received over a rolling window.
• 99% of decoded events complete within the workload-specific budget.
• Eligible-intent landing rate stays above the route-specific floor.
• Duplicate execution remains zero.
• Clock offset remains below the benchmark uncertainty budget.

Choose numbers from measured baselines and business tolerance. Do not copy a vendor SLA into a strategy SLO without testing whether it predicts fills.

Production checklist

Before trusting the dashboard:

• Create separate opportunity, intent, and signature identities.
• Use monotonic clocks for local duration math.
• Monitor wall-clock offset on every host.
• Store commitment transitions as append-only events.
• Keep ACK, first-seen, processed, confirmed, and finalized separate.
• Define landing eligibility before calculating the denominator.
• Record fee, tip, route, retry, and blockhash age.
• Use bounded metric attributes.
• Put signatures and wallet addresses in protected logs or traces, not labels.
• Measure loss, gaps, and missing observations beside latency.
• Reconcile every provisional action.
• Connect technical metrics to realized execution value.

The takeaway

Solana trading observability is a correlation problem. The system must join an early signal to one business intent, every signed attempt, independent network observations, commitment transitions, and the final economic outcome.

Instrument the boundaries once. Keep exact evidence in logs and traces, distributions in metrics, and high-cardinality identifiers away from metric labels. The result is a dashboard that explains missed fills instead of merely proving that an endpoint answered.