xdrcheck replay against their indexer and need to read the result: how
ledgers are served, how a run is planned, how stored rows are matched and classified, and what the report and exit
codes mean.
The harness feeds corpus ledgers to an indexer through an input the indexer already reads, drives it through an
adapter (adapter-protocol.md), compares what it stored with the manifest
(manifest.md §6) and classifies every entry. Commands and flags: cli.md.
1. Shape of the harness
xdrcheck serve is the ledger server on its own, for pointing any indexer at the corpus by hand. xdrcheck replay plans,
serves, drives an adapter, matches, classifies and reports.
2. Ledger server
All inputs are views over the same ledgers. The server never alters a ledger’s bytes except in the stream unit (§3.3), which is a declared transformation.2.1 SEP-54 layout on disk — supported first
Exactly what go-stellar-sdk’ssupport/datastore reads and Galexie writes: a root .config.json (the compact
JSON the SDK marshals, with the pass’s passphrase, ledgersPerBatch: 1 and batchesPerPartition: 64000) and
one zstd-compressed LedgerCloseMetaBatch per ledger at the key the SDK derives from the sequence, for example
testnet ledger 4437357 at FFBC9DFF--4416000-4479999/FFBC4A92--4437357.xdr.zst. The batch is written by
concatenation (uint32 S ‖ uint32 S ‖ uint32 1 ‖ <LedgerCloseMeta bytes>), so corpus bytes are never re-encoded.
A history entry is served as the original public-lake object, byte for byte, at its original key, after its
sha256 is checked. Every SDK backend demands strictly consecutive sequences, so a layout is always gap-free.
This reaches go-stellar-sdk’s BufferedStorageBackend with a filesystem datastore and any other lake reader.
2.2 The same layout behind an object store
Pipelines hard-code their store type (wallet-backend reads S3; stellar-etl reads GCS), so the layout is also served over an object-store API. For S3 the harness runs SeaweedFS — the emulator wallet-backend’s own tests use — from the image digest pinned inpins.toml (never pulled during a run), uploads each unit under its own
prefix, and puts a small logging reverse proxy in front of it; consumers are pointed at the proxy, whose request
log is the delivery evidence (§2.5). Where no container can be created, the image’s own statically linked weed
executable runs as a local process instead (same digest). Every port is bound on loopback only.
GCS is not served by xdrcheck yet (--input sep54-gcs and xdrcheck serve --gcs exit 2). The stellar-etl adapter takes
the sep54-dir layout and runs its own GCS emulator (fake-gcs-server, the one go-stellar-sdk’s datastore tests
use).
2.3 Mock getLedgers — supported second
JSON-RPC 2.0 over HTTP on loopback, answering getHealth and getLedgers with stellar-rpc’s wire types (the
go-stellar-sdk protocols/rpc structs stellar-rpc itself uses), so Go, JavaScript and other clients decode it
unchanged. Rules copied from the real server: parameters by name only; getLedgers limit default 50, maximum
200; a cursor together with a non-zero startLedger, a start outside the served range, a limit over 200 or a bad
xdrFormat are error −32600 with the real message text; a bad cursor is −32602. metadataXdr is the corpus
LedgerCloseMeta verbatim; headerXdr is the embedded LedgerHeaderHistoryEntry re-encoded with the pinned XDR
library.
Deliberate differences: the mock is always healthy (real RPC reports unhealthy when its latest ledger is more than
30 s old, which would stop every client replaying old ledgers), and xdrFormat: "json" is not served.
getLatestLedger, getNetwork and getVersionInfo are answered too; every other method gets a well-formed
−32601 error and is logged, because some clients call other methods and carry on after the error.
2.4 What known clients request
No SDK ledger backend checks the previous-ledger hash or that a ledger’s own sequence equals the requested one; all demand strictly consecutive sequences, and all enforce the passphrase indirectly by re-hashing envelopes. After the last served ledger a lake backend retries the next key and an RPC backend pollsgetHealth, so an
adapter decides completion from the indexer’s progress marker (cursor ≥ range.end), not from the indexer
exiting.
2.5 Delivery evidence
To tell an indexer that hangs from a ledger that never reached it, each view records what it served: the mock RPC logs every request, and the S3 proxy logs every object request (uploads and the harness’s own read-back check go to the store directly and are not delivery records). Reads of asep54-dir layout are not observed. The
classifier uses this evidence (§7, row 0); xdrcheck serve --delivery-log FILE writes it for hand runs.
3. Replay units
3.1 single
The server publishes exactly one ledger. The adapter sets the indexer’s start toS (for wallet-backend, by
seeding its cursor to S − 1). Entries that share one ledger share a unit. An entry that needs earlier ledgers
sets replay.requires_prior_state: true and is never served outside its chain.
3.2 chain
A scenario run starts at genesis on a fresh network, so everything a later ledger depends on is in the chain. Its ledgers, passphrase, protocol and settings come from its chain index (corpus/chains/<scenario>/…json,
manifest.md §8.2). All entries produced by that run are scored from one session. If the indexer
stops at ledger K, entries before K are scored from their facts, the entry at K takes the stop’s class, and
entries after K are not_reached.
3.3 stream — the exact transformation
For the i-th ledger of the stream (base sequenceB, default 1000, --stream-base): the header’s ledgerSeq
becomes B + i; every lastModifiedLedgerSeq inside the meta that equalled the original sequence becomes B + i,
and so does an account’s seqLedger when it equalled it. Nothing else changes, so transaction hashes are kept
while the ledger hash and the hash chain go stale (no SDK backend checks them). TTLs, time bounds and other ledger
arithmetic keep their original numbers. This mirrors go-stellar-sdk’s ingest/loadtest. Stream bytes are never
written into the corpus, and stream results are never used for known-divergence assertions or findings: a
failure first seen in a stream must be reproduced as a single or chain.
4. Planning a run
- Select
ledgerentries whosereplay.indexer_scopeis notsdk-only, filtered by--entries,--capsand--network. - Split by passphrase: one pass per network; each variant is planned in the pass of its network.
- Apply the envelope from
describe, using manifest facts only, never decoding: withsdk_limit(or--out-of-envelope skip) an out-of-envelope variant is not run and isnot_applicable; withindexer_gateit is run, to probe the gate. - Choose units: single by default; chain when the entry asks for it or
--unit chainis given; stream only with--unit stream. - Choose the input:
--input, else the first ofsep54-s3(only when the pinned store image is present locally),sep54-dir,rpcthat the adapter declares. - Order units deterministically: pass by network, then unit kind, first served sequence, first entry ID.
- Size deadlines from the adapter’s
describe.timing, unless--deadlineis given.
5. Running
Per unit, on one worker: publish the unit on a fresh ledger view →prepare → ingest → observe (always,
whatever ingest returned) → teardown → match → classify → judge against known divergences → write one row per
entry variant.
- Isolation. A fresh session, a fresh ledger view and its own delivery evidence per unit. Each worker
(
--parallel N, capped by the adapter’smax_parallel_sessions) has its own adapter process; an adapter that breaks the protocol, does not answer or dies is killed and a fresh one is started for the next unit. - Limits. The adapter’s group runs under
--memory-maxand--pids-max(cgroup v2 scope, or an address-space limit with resident-memory polling where no user cgroup is available; the report names the mechanism). Deadlines as in adapter-protocol.md §6. - Bounded memory. After
ingesta unit is observed in windows of consecutive ledgers, and each window’s entries are matched, classified and written before the next window is requested, so the harness holds one window’s observation set at a time. Adapter response lines are capped at 64 MiB. - Confirmation.
--confirm Kre-runs a unit with a non-pass result K more times; results that differ areflaky. - Network. The harness opens only loopback sockets. Run under
make … OFFLINE=1(orscripts/offline-run.sh) the whole run, adapter and indexer included, has no route out.
6. Matching observations against expect
The matcher implements manifest.md §6, which is normative. In short:
- Exact comparison everywhere. Amounts, IDs and counts are compared as decimal strings or integers; no tolerance, rounding, case folding or prefix matching.
- Stored values are decoded by the harness with its own strict strkey decoder, according to the declared
as_stored.encoding; an adapter’s interpreted fields are used only where no stored form is given. - Scope. Facts belong to the focus transaction or to the transaction an item names; other transactions are looked at only through the receipts and the transaction list. Ambient facts (balance changes, sequence bumps, fee and refund movements) satisfy an item that lists them and are never extras.
- Presence. Kinds and sub-kinds the adapter does not provide are
not_applicable; anonly_ifitem is required only when the adapter declares that modelling choice. - Declared scope. An expected participant, stored address or state change that meets a rule of the
capability’s
out_of_scopelist (by address type, role, the asset type of a participant’s movements, kind, asset type, detail or cause; each rule cites the indexer’s own statement asbasis) isnot_applicablewith that basis as its reason (adapter-protocol.md §8.1). Negative facts and extras are never excused; a malformed list makes the match fail. - Identity. Participants match by base address. A
base_onlyreader’s muxed ID is never compared; abase_and_aliasreader must report the expected one. Identity held only in muxed form is a wrong value. - Negative facts (
must_not_include, acount: 0movement) are checked only when the transaction is positively observed, and a result made of negative facts alone is never a pass.
matched, missing, wrong or not_applicable; observed facts no expectation accounted
for are ignored, extra (under an exactness flag) or forbidden.
7. Classification
The first matching row gives the class and qualifier. “This ledger” for a stop is the ledger the indexer stopped on: the ingest result’sfailing_ledger when given; else the one after its cursor; for a stop the harness
observed itself (deadline, memory kill), the last ledger the delivery evidence shows served.
The report has three result classes, pass, mishandled and dropped, plus four side categories
(
not_applicable, not_reached, flaky, harness_fault) that say an entry produced no result about the indexer,
and why. A verified refusal (dropped · refused) is the safe behaviour the guide recommends: it needs no finding
and does not fail a run. A refusal the harness cannot confirm is already a halt (row 5).
8. Report
With--report DIR a run writes:
8.4 Exit codes
9. Known divergences: how they are asserted
An entry’sconformance.known_divergences may name an indexer with level: "replay"
(manifest.md §7). For a run against target T at version V:
- Divergences whose
targetis T and whoseversionsrange contains V apply. For packages with tags the harness evaluates the range; for repositories without tags the adapter answers throughprepare.resolve_ranges. - Inside a range the result must be the declared class and, when given, qualifier. Anything else, a pass included, is unexpected: the corpus no longer reproduces the defect.
- Outside every range the result must be
pass. xdrcheck replay --assert-knownruns only the entries with a divergence for T and fails on any unexpected result.
make verify-replay (scripts/verify-replay.sh) runs --assert-known with the wallet-backend adapter at each
pinned pre-fix and fixed commit of wallet-backend#729, #731 and #716 (the acceptance key of
[pipelines.wallet_backend] in pins.toml) and at the pinned current commit, on both the S3 and the RPC input. The
entries are found in the manifest, not listed in the script. Besides the exit code it asserts row by row:
The hand-run reproductions these checks were built from are in
evidence/wallet-backend-replay/ and
evidence/wallet-backend-731/.
10. The memory-bound test
make verify-memory (scripts/verify-memory.sh) runs four tests, each xdrcheck replay in a memory-limited group of
its own (256 MiB, swap off):
A negative control then reruns T-mem-1 with streaming switched off and must fail, so the test can tell a streaming
harness from one that is not.
A run’s peak is the xdrcheck process’s peak resident set (
VmHWM), which includes about 37 MiB of the binary’s own
mapped code. It is read only once the process runs xdrcheck: until the exec through systemd-run completes, the new
process can still show the test’s own memory, which once was recorded as the run’s. The rest is bounded three ways,
so the peak does not depend on the machine’s CPU count or on the collector’s timing: the harness compresses ledgers
with one zstd encoder rather than one per CPU (each holds about 16 MiB of history); an adapter line is collected in
fragments and joined once, so a line cut at the cap never costs twice its length; and before each slice of work the
Go soft memory limit is set to what the harness holds plus room for the slice, at most 64 MiB, so a stream of
ledgers of megabytes does not grow the heap up to the ceiling taken from the group (87.5 % of it).
11. Command-line surface
See cli.md for every flag ofxdrcheck serve and xdrcheck replay, generated from their help output.
12. The harness’s own test: seeded-bug indexers
make harness-selftest (scripts/harness-selftest.sh) replays the corpus through the baseline indexer with one
deliberate defect switched on at a time (baseline --bug NAME) and requires each to land in the class its row of
go/adapters/baseline/bugs/table.json documents; the unmodified baseline must pass the whole corpus. Together the
bugs reach every classification branch: truncated or payload-dropped addresses, fabricated muxed addresses,
participants taken from the authorization tree or the envelope, identity kept only in muxed form, credit from a
forged token event, double-counted transfers, float amounts and void routing read as zero
(mishandled · wrong_value); a silently dropped ledger (dropped · silent); a dropped contract trustline
(dropped · partial); a panic (mishandled · crash); an endless loop (mishandled · hang); permanent-error exits,
a partial write, fake and dirty refusals (mishandled · halt); a genuine protocol gate (dropped · refused); extra
fee rows (pass: storing more is not a failure); and a memory hog.
13. Service rigs (xdrcheck replay --rig)
xdrcheck replay --rig wallet-backend|stellar-etl --ledgers A-B|A,B,C|@FILE --lake DIR --commit SHA --out DIR [--network pubnet|testnet|PASSPHRASE] [--shards N] [--log-level LEVEL] runs the real pipeline, built from source at the commit (cached
per commit), over ledgers read from a lake-layout directory. Ledgers the directory lacks are fetched from the public
lake unless --fetch none. The run directory holds manifest.json, status.jsonl (one line per requested ledger:
ingested, halted with the error text, the ledger it names and the durable cursor, skipped, or harness_fault),
rows.jsonl (the judge’s row kinds, sorted by ledger), rows-other.jsonl (every other row, so nothing written is
left out), checks.jsonl and summary.json; each shard keeps its logs and raw tables or exports.
wallet-backend:
- Contiguous ledgers form a segment. Each segment is one live-ingestion process into the shard’s single database,
with one ledger per commit; the cursor is polled, so every commit is observed. Before a segment the rig sets the
ingest cursor and the history and current-state cursors of every classified protocol to the segment’s first
ledger minus one, which is what a finished
protocol-migrateleaves. Live mode cannot cross a gap, so a sparse list is a series of segments in one database; SEP-41 running totals then start from zero at the first segment. - A segment that stops short: the ledger after the durable cursor is
halted, the service is started once more without touching the cursor to see whether the halt repeats, and the rest of the segment isskipped(--skip-haltedmoves the cursor past the halted ledger instead, which production never does). - Seeding, in place of the history-archive checkpoint that a seeded cursor skips: a
contract_tokensSAC row for every Stellar Asset Contract the ledgers touch (verified by hashing the asset, written as the checkpoint writes it) unless the ledgers deploy it; and, from a state seed (--seed DIR, built once with--seed-rpc URL, with an optional request header from$XDRCHECK_STATE_RPC_HEADER), theprotocol_wasmsandprotocol_contractsrows of the WASM contracts the ledgers touch. A contract whose instance a replayed ledger changes is bound to the WASM of that change’s pre-image (its binding at the time); any other contract is bound as the seed has it, which is the network’s state when the seed was fetched, not at the replayed ledger. Every WASM an instance in the ledgers points at is registered and its code served. Nothing is pre-registered (no SAC row, WASM row or binding) for a contract deployed or a WASM uploaded inside the replayed ledgers: live ingestion sees those happen and records and classifies them itself. The commit’s ownprotocol-setupthen classifies them against a local RPC that serves the seed’s entries. - Rows:
transactions_accountsgiveparticipant;operations_accountsgiveop_participant; every balance state change givesbalance_changenamed by its token’s contract (fee rows marked; the judge tells verified SACs from other contracts); the other state changes giveentry_changerows (account, trustline and data entries created or removed, and the signer, threshold, flag, home-domain, limit, authorization and data-value changes they state, signer keys as stored) beside the verbatimstate_change; balance tables givebalance(absolute state, not deltas). - The local RPC answers
simulateTransactiononly for a zero-argument contract call, with the return value of the same call recorded in the run’s ledgers; anything else is a failed simulation.
export_* command the binary lists, over each segment as one range; a one-ledger segment N is
exported as N..N+1 (rows of N+1 are context and are dropped), because a one-ledger range writes no ledger entry
changes. Each requested ledger is checked against what it holds (ledger, every transaction and operation, a fee
transfer per transaction, contract events, changed entry keys); a failed check is a silent drop. Rows: token
transfers give token_event and the balance_change rows they imply; effects give effect_balance_change
(amount-bearing effects) and entry_change (entries and attribute values the effects state); an operation’s
asset_balance_changes give op_token_event.
Baseline adapter
For indexer authors who want to see what a correct reader extracts from aLedgerCloseMeta, or who are adding
observation kinds to their own adapter. The baseline is a minimal ledger reader built on go-stellar-sdk v0.9.0, plus
the adapter that lets xdrcheck replay drive it. It has two jobs:
- Reference reader. It shows what a correct reader extracts from a
LedgerCloseMeta, and from where. Every replayable corpus entry must match it with no missing or wrong facts. When it fails, either the baseline or the entry is wrong, and the entry is blocked until that is settled. - Complete example adapter. It maps every observation kind, so it shows how each fact is derived and reported.
To adapt your own indexer, start from the template
../minimal(receipts and transactions, about 150 lines) and come here for the kinds you add. Both are built on the public packagego/adapterkit, which holds the protocol types, the stdio transport and the test helpers; neither the template nor this adapter’s protocol code imports anything underinternal/. The contract stays the two schemascorpus/schema/adapter-protocol.schema.jsonandobservation.schema.json, so an adapter in another language needs nothing from this repository (examples/adapter-ts).
Build and run
baseline [--max-protocol N] [--max-lcm-version N] [--min-protocol-with-transactions N]is the adapter. It speaksxcc.adapter/1: one JSON request per line on stdin, one response per line on stdout, logs on stderr.baseline index …is the indexer. The adapter starts it for eachingest.
describe always declares exactly the limits the
indexer enforces.
What the indexer extracts, and from where
Events are trusted only as far as their emitter (rule M-7 in
docs/manifest.md §6.7):
- An event is a movement of a classic asset only when the emitting contract is that asset’s Stellar Asset Contract under the pass’s network.
- An event from any other contract that names a classic asset in its topics is a forged claim. Nothing is recorded, and the address it names does not become a participant.
- An event that names no classic asset is that contract’s own token movement.
indexer/derive.go and indexer/events.go. Each step is a separate function (feeStep,
txLevelStep, opChangesStep, eventsStep, balanceDeltaStep, aliasStep), so a seeded-bug variant can
replace one step and nothing else. The envelope’s claims (source, destinations, memo) are stored beside the facts
under claimed and are never mapped to facts.
Gates (enforcement: indexer_gate)
A gate is checked before anything is written for the ledger. A refusal:
- exits with code 3;
- leaves
exit.jsonnaming the gate and the observed value; - leaves the cursor below the refused ledger.
- a fetch, decode or persist error exits with code 1 and an
exit.jsonnaming the phase; - a Go runtime panic exits with code 2.
The store
The indexer has no database. Each session’s store is<workdir>/store/:
ledgers/<seq>.json: one document per ledger, holding its facts in the protocol’s neutral vocabulary (one record per fact) and theclaimedblock;cursor: the last ledger stored;exit.json: written when a run stops on a ledger.
observe reads the documents back and sorts the facts by transaction index, operation index and JSON text. The
indexer reads no clock, so two sessions over the same unit give byte-identical observation sets.
Writing your own adapter
Copy../minimal and follow its README. When your indexer is a separate program with a store of
its own, as here, this adapter’s protocol handler (adapter.go, on adapterkit.Serve) is the pattern:
describe: keepDescribeconstant. Declare only the inputs, units and observation kinds your indexer really has. Declare its real gates, orenforcement: none.prepare: create an empty store for the session. Refuse a store you did not create.ingest: start your indexer against the loopback input and range you are given, with a scrubbed environment. Read its progress marker and its own structured exit record, never its log text. Report the facts behind the status (ingestResultshows the order of the rules).observe: map one stored record to one fact. Pass stored values through asas_stored, and do not clean them.teardown: remove the session’s store.- Check it: run
xdrcheck replay --adapter "<your command>" --selftest.
adapter.Selftest, then plan → serve → adapter → match over every
replayable corpus entry, on the SEP-54 directory, mock RPC and (where the object store can run) S3 inputs.
stellar-etl adapter
For stellar-etl operators and for authors of batch-export indexers: how the stellar-etl adapter builds and drives an unmodified stellar-etl binary at the version under test throughxcc.adapter/1, how it decides a status without a cursor, and how export rows become facts. Executed results:
evidence/stellar-etl/.
Build and run (stellar-etl adapter)
--target-version takes a tag (v2.8.35, the release pins.toml [pipelines.stellar_etl] pins) or a full commit.
Options of the adapter itself:
What it does per session
prepareresolves the target in a blob-less clone of stellar-etl (<cache>/stellar-etl/repo.git, cloned on first use), buildsbin/stellar-etl-<commit>once with the Go toolchain the commit’sgo.modnames and cgo on (stellar-etl links go-stellar-xdr-json, a cgo wrapper), and answersresolve_ranges:A..Bis every commit from A to B on B’s first-parent history,..Beverything up to B. A cold build needs the network once (the clone, Go modules, the toolchain); later runs work offline.ingestserves the harness’ssep54-dirlayout through a GCS-compatible emulator on loopback (fake-gcs-server’s library, the emulator go-stellar-sdk’s own datastore tests use) and runs every export stellar-etl has —export_ledgers,export_transactions,export_operations,export_effects,export_token_transfer,export_contract_events,export_trades,export_ledger_entry_changes,export_assetsandexport_ledger_transaction— over the range, one after the other (a failed export does not stop the ones after it), each with-o <session>/out/<command>/ --datastore-path xdrcheck/lakeand the network flag (--testnet,--futurenet, none for pubnet), in a scrubbed environment (PATH,HOMEandSTORAGE_EMULATOR_HOSTonly).observereads the JSON Lines back and maps them (below).teardownremoves the session’s folders.
sep54-dir is the only input declared; the emulator is the adapter’s, and the indexer is
pointed at it exactly as it would be pointed at a bucket. Other passphrases than pubnet, testnet and futurenet cannot be configured in stellar-etl and are
answered with unsupported_input.
Status
stellar-etl is a batch tool with no cursor. The adapter’s progress marker is the highest ledger in the ledgers export once every command has exited 0; status rests on exit codes, signals and rows, never on log text.Envelope
sdk_limit, from the go-stellar-sdk version the target’s go.mod pins: v0.8.x (including the v0.8.1
pseudo-versions of v2.8.33 and v2.8.34) → max_protocol 30; v0.7.x → 29. Without a local clone at describe time,
or for any other SDK version, nothing is declared (enforcement: none). The pinned v2.8.35 pins go-stellar-sdk
v0.9.0, which the adapter does not classify yet, so at the pin no envelope is declared.
Row mapping
One stored row is one fact; nothing is derived or repaired.export_ledger_entry_changes, export_assets and export_ledger_transaction are mapped onto no fact: they are
judged on how they end (status table above, and the halt oracle). scripts/replay-etl.sh (XDRCHECK_REPLAY_ADAPTER=stellar-etl scripts/replay-full.sh) replays the whole corpus with the halt oracle on and fails on any export that did not end ok.
Effects and operations carry no transaction hash; their TOIDs (ledger<<32 | order<<12 | op) locate them through
the transactions export.
Service rig (xdrcheck replay --rig stellar-etl)
- Network and path. stellar-etl has no passphrase flag:
--testnet/--futurenet/ none pick the passphrase and the datastore suffix (<--datastore-path>/<testnet|futurenet|pubnet>); the layout is fixed at one ledger per file, 64,000 files per partition. Standalone ledgers closed with the testnet passphrase are exact under--testnet. A wrong network flag still exits 0 with rows (transaction hashes then come from the wrong passphrase), so the rig derives the flag from the lake’s passphrase and refuses others. - Absent ledgers are fatal in stellar-etl (exit 1, “ledger object … is missing”), not an empty export. The rig passes an absolute output directory (each export runs with its working directory in the segment folder) and calls a command that reports successful transforms but wrote no file a harness fault.
- Status.
ingested,halted,skipped,harness_fault, anddropped: every export exited 0 but a drop check failed (drop_checks_failednames them). Nonce entries are left out of the contract-data check (stellar-etl discards them by design). - Judge rows.
ledger,token_eventandbalance_changego to the judge: balance changes are derived fromexport_token_transferrows of classic assets (fees included; use--balance-scope net), never from effects, which have no credit or debit for trades, pool legs, creation and merges. Pass--kinds ledger,balance_change,token_event: entry changes are compacted per ledger without a transaction (ledger_entry_change), transactions give only source and fee account, andapproveis not exported.
Tests
observe_test.go— the mapping on rows stellar-etl wrote for ledger 64558277 (testdata/out), and the harness’s matcher on them: the mint in that ledger is a wrong routing value.gcs_test.go— go-stellar-sdk’s GCS datastore, the reader stellar-etl uses, reads a layout from the emulator.adapter_test.go— the protocol selftest, completed and halted ingests, the passphrase refusal and commit ranges, with a stand-in for stellar-etl that reads the ledger from the emulator and writes the fixture rows.real_test.go— withXDRCHECK_STELLAR_ETL_CACHEpointing at a cache that holds the clone and the binary of the pinned commit: the real stellar-etl over every entry declaring a stellar-etl replay divergence (--assert-known); skipped otherwise.
Horizon rig
For Horizon operators and contributors: how to run real ledgers through an unmodified stellar-horizon, export what it stored as rows the judge reads, and what the rig cannot see. Horizon is fed by its own history reingestion with the datastore ledger backend, so no captive core, no history archive and no network are involved.Build and run (Horizon rig)
horizon-rig rows --out rows.jsonl [--commit C] RUN/tables re-exports the common rows from a run’s raw dumps.
What it does
- Starts Postgres and one long-lived container for the static Horizon binary on an internal Docker network (no
egress), creates one database per shard and runs
db initin each. - Ledgers whose file is not in the lake are
skippedand never sent to Horizon. The rest are planned into segments and spread over the shards. Each segment is onestellar-horizon db reingest range S E --ledgerbackend datastore --datastore-config <Filesystem datastore> --network-passphrase P --history-archive-urls <unused> --db-url <shard>. - A segment that exits 0 marks its ledgers
ingested. A failing single-ledger segment ishaltedwith the error Horizon logged. A failing multi-ledger segment commits nothing and its error need not name the ledger, so every ledger of it is rerun alone: each becomesingestedorhaltedon its own (attempt: probe), andstopped_atkeeps the last ledger the backend returned before the segment stopped. - Every status is cross-checked against the shard’s
history_ledgers; a mismatch is a harness fault (exit 3), and so is a failure of the rig’s environment before Horizon reads the ledger (statusharness_fault). History-archive caching is off: concurrent calls in one container otherwise race on Horizon’s shared bucket-cache directory. dump.sqldumps every history table Horizon fills, with internal ids resolved to addresses, assets, pool and claimable-balance ids, intotables/<table>.jsonl.zst;rows.jsonlmaps them to the common rows.
Run directory
Common rows
Rows are what the judge (crates/xdrcheck-judge, verdict --rows) reads, plus the rig row fields: kind,
pipeline (horizon), commit, source (the table), ledger, tx_index / op_index (0-based apply order;
Horizon’s are 1-based), tx_hash (outer hash for a fee bump), exact (always true), amounts as integer strings in
the smallest unit. Run the judge with --balance-scope nofee and --kinds limited to what Horizon exports.
Limits
db reingest rangeruns Horizon’s transaction (history) processors only. The state tables (accounts,accounts_data,accounts_signers,trust_lines,offers,liquidity_pools,claimable_balances,claimable_balance_claimants,asset_contracts,contract_asset_balances,contract_asset_stats,exp_asset_stats) stay empty; they need history-archive checkpoint ingestion, which this rig does not run. The 1-minute trade aggregationhistory_trades_60000is filled and dumped (asotherrows).- History reingestion needs no neighbours: a ledger ingested alone gives the same rows as the same ledger inside a dense range, so sparse ledgers are fine.
- Horizon has no fee effects, so its balance changes never include a transaction’s fee (
feeis always false). - Horizon emits
account_credited/account_debitedfor an issuer moving its own asset; the judge holds no balance for an issuer, so those rows read as extra. A path payment whose source is also a maker of one of its hops loses that maker side. - A Horizon release stops on a ledger whose protocol is newer than the one it supports; that is a
haltedstatus with Horizon’s message, not a harness fault.