xdrcheck replay drive one indexer. After
this page you can implement every message, report every ingest outcome correctly, and pass --selftest.
The message shapes are fixed by two JSON Schemas (2020-12): corpus/schema/adapter-protocol.schema.json and
corpus/schema/observation.schema.json. The messages quoted below are in corpus/schema/examples/adapter-protocol/
and validate against them. Where this page and the schemas disagree, the schemas win. How the harness plans, matches
and scores a run is in replay-harness.md.
1. What an adapter is
The harness knows nothing about any particular indexer. An adapter is a small program that knows one. On request it puts the indexer in a clean state, makes it ingest a range of ledgers from an input the harness serves, and reports what the indexer then holds, in a neutral vocabulary. An adapter never judges. It reports machine-observable facts; the harness compares them with the corpus manifest and does all scoring. Process signals alone cannot tell a bug halt from a deliberate stop on unsupported input, so the decision combines the adapter’s facts with what the manifest says about the entry.go/adapters/walletbackend (the reference adapter),
go/adapters/stellaretl, go/adapters/baseline (a minimal correct indexer and the complete example adapter;
see its README), and go/adapters/minimal, the template for new adapters (§8).
2. Transport and framing
3. Versioning and capability negotiation
- The protocol name carries a major version. Within
xcc.adapter/1changes are additive (new optional fields; new enum values only where the schema leaves a field open). Adapters ignore unknown request fields; the harness is strict about responses wherever the schema forbids additional properties, so typos are caught. describeis always the first call. The harness lists the protocol versions it speaks; the adapter answers with one, or with the errorunsupported_protocol_version.- Capabilities are declared, not discovered.
describestates the inputs the indexer can read, the replay units the adapter can run, the observation kinds the indexer models (withsubkindsandmodellingchoices), and the envelope: the protocols and meta versions the indexer at this version claims to handle. The harness plans the run from these declarations and records them indescribe.jsonnext to the report.
3.1 The envelope, and why “refused” is narrow
A deliberate fail-closed stop is safe and is not scored like a crash. A halt on valid input is a bug. From outside the process the two look the same, so the adapter declares its limits up front:
Rules the harness enforces whatever the adapter says:
- A stop on an entry inside the envelope is a halt (
mishandled · halt). An adapter cannot turn it into a refusal. refusedis accepted only if the gate named in the response was declared indescribe, the manifest confirms the entry really is outside that gate, the indexer wrote nothing for the refused ledger, and the process did not crash.- Gates are a closed list, each with a predicate the harness checks against the manifest:
min_protocol,max_protocol,min_protocol_with_transactions,ledger_close_meta_version,transaction_meta_version,operation_type.
4. Messages
Five methods, in this order within a session:describe (once per adapter process) → prepare → ingest →
observe → teardown. A session is one replay unit: one fresh indexer state, one ledger range. Messages are
shown wrapped where long; on the wire each is one line.
4.1 describe
timing sizes the harness’s ingest deadlines. participants.muxed is base_only (the indexer stores the base
account) or base_and_alias (it also stores the muxed form beside it). address_encoding says how stored
addresses are passed (strkey_text, version_byte_payload_hex, xdr_hex, raw_key_hex).
4.2 prepare
Puts the indexer at target_version into a clean state for one session. It may reuse anything in cachedir
(built binaries, database templates) but nothing from another session’s data.
resolve_ranges exists because some targets have no release tags: the manifest’s known divergences then name
commit ranges, and only the adapter holds the repository that decides whether the version under test lies inside
one (A..B is every commit from A to B on B’s first-parent history, ..B everything up to B). For tagged
packages the harness evaluates ranges itself and omits the field. (artifact_sha256 above is a placeholder.)
4.3 ingest
Makes the indexer ingest range from input and returns when it has finished the range, has stopped, or the
deadline has passed.
Input descriptor
range is inclusive and names real ledger sequences as served. network.passphrase is the passphrase of every
ledger in the range: one harness pass per passphrase, because readers verify transaction hashes against it.
schema mirrors the store’s .config.json (ledgersPerBatch, batchesPerPartition). The dummy credentials
matter: without them an AWS client probes the instance-metadata address, which is an outbound request.
Ingest statuses and the facts that justify each
The adapter chooses the first row whose facts hold.stderr_tail is for the report only; no status may rest on
matching log text.
last_ledger is always the indexer’s own account of how far it got. rows_for_failing_ledger lets the harness see
partial writes.
4.4 observe
Returns what the indexer holds for a range as an observation set (xcc.observation/v1). It is called after
ingest whatever the status, so partial progress is visible.
7844ad…879b20 — 33 bytes, the signed payload dropped — and nothing else differs. The harness decodes
the stored bytes itself and finds they are not the expected P… key.
Observation rules
- Report what is stored, exactly. Every address-bearing value has an
as_storedform: the literal text or bytes the indexer keeps. Interpreted fields (address,signer.key,on_chain_form) are filled only when the indexer itself holds that form. An adapter never computes a base account from a muxed one, re-derives a strkey, pads, trims or rounds on the indexer’s behalf: that would hide exactly the bugs under test. - Amounts and 64-bit IDs are decimal strings. A value that is not an exact integer in the indexer (a JSON
float, a decimal with a fraction) goes literally into
amount_as_stored;amountis left out. - One stored record is one fact. An adapter maps each record to at most one fact of the closest neutral kind,
never merging or splitting records or filling a field the record lacks. Differences between indexers’ models are
absorbed by the declarations in
describe(subkinds,modelling,muxed), not by the adapter. An indexer that writes one balance row per event (per mint, transfer or path hop) is reported row by row; the harness compares those rows with the reference’s net change by their sum (manifest.md §6.6, S-7). - State-change
detailkeys are closed per kind, the same list as the manifest’s (manifest.md §6.6), enforced by the observation schema. A value the indexer holds in another representation goes under<key>_as_storedas the literal token, which is evidence only. - Result codes use XDR-JSON names (
tx_success,tx_failed, …) mapped from the indexer’s labels through a fixed table; the original may be kept innative_label. - Kinds are optional; absence is declared. Each kind is
providedornot_applicablewith a reason. An expected fact whose sub-kind is not listed isnot_applicablefor that fact. A provided kind that comes back empty means the indexer holds nothing — that is how drops are seen. A ledger missing fromledgerswas not seen. - Locators. Facts carry
tx_hashwherever the indexer can supply one, andtx_index/op_indexwhen it has them. Facts without a transaction locator are matched at ledger scope.
4.5 teardown
keep_state: true (xdrcheck replay --keep-state) the adapter leaves everything in place and lists it in
left_behind.
4.6 Errors
An error response means the adapter or its environment failed; it is never a statement about how the indexer handled a ledger. The harness records the affected entries asharness_fault.
5. Determinism
- No wall-clock values in results except
wall_ms, a duration used for reporting and deadline sizing, never for scoring. - Arrays in an observation set are ordered:
ledgersbyledger_seq; within a ledger, facts by transaction, then operation, then the fact’s JSON text. Two runs on the same input give byte-identical observation sets oncewall_msis removed. - The adapter keeps the system clock out of observations: where the indexer stamps rows (
ingested_at), that column is left out. - Session IDs, ports and directories come from the harness; the adapter invents none.
6. Resource limits — owned by the harness
-
Time. Deadlines per call:
-
Response size.
xdrcheck replayrefuses a response line longer than 64 MiB (aharness_faultfor that unit) and asks for observations in windows of consecutive ledgers, so an adapter never returns a long unit in one line. -
Memory and processes. The harness starts the adapter as the leader of a new process group inside its own
cgroup v2 scope (
MemoryMax, swap off,TasksMax;xdrcheck replay --memory-max,--pids-max), so everything the adapter spawns is inside it. Where no user cgroup is available it falls back to an address-space limit plus polling of resident memory, and the report says which mechanism was used. An out-of-memory kill of the indexer surfaces ascrashed; one of the adapter itself is recorded by the harness asmishandled · crash. -
Services an adapter runs in containers are outside that group. The adapter receives the limits in
prepare.limitsand passes them on.
7. Security rules
- No network. Adapter and indexer must work with outbound networking disabled;
make verify-replay OFFLINE=1runs the whole harness in a network namespace with loopback only. Every URL in a request is a loopback address (the schema enforces it). The adapter sets every endpoint of the indexer explicitly, because defaults may point at public infrastructure. - No credentials. The protocol carries no secrets; object-store credentials are dummy values for the local endpoint. The indexer is started with a scrubbed environment.
- No production data. An adapter works on a fresh store it creates in the session.
- No disclosure required. Nothing in the protocol needs the indexer’s source, schema or internal names;
fieldandnative_labelare optional, so an adapter for a private indexer can leave them out.xdrcheck report redactstrips evidence strings from a report before it is shared. - Corpus bytes are untrusted input from the indexer’s point of view (some entries are built to stress readers). Adapters do not pre-validate or filter them.
- Telemetry in dependencies is switched off (the wallet-backend adapter disables TimescaleDB’s telemetry job).
8. Writing an adapter
Start from a template:- Go:
go/adapters/minimal— about 150 lines, receipts and transactions, every place to change markedREPLACE. It is built on the public packagego/adapterkit(github.com/OWNER/REPO/go/adapterkit, standard library only): the message types,Main/Serve(the stdio transport, the version check, error responses, the refusal of undeclared inputs and units),RunChild(an indexer process with a deadline),ProvidedKindsandSortFacts, the embedded schemas with the harness’s own response check, andkittest, the selftest’s checks in process for your owngo test. Copymain.goandprotocol_test.gointo your own module; nothing underinternal/is needed (a test builds the copy as a module of its own). - Any other language:
examples/adapter-tsis the same template in TypeScript. The contract is only the two schemas,corpus/schema/adapter-protocol.schema.jsonandobservation.schema.json. - Every kind:
go/adapters/baselineis the complete example: an indexer that is a separate process with its own store, and all six kinds.
scripts/demo.sh runs the Go template with one planted bug and prints what the report says about it.
For any indexer:
- Choose the input. An indexer that reads a data lake through go-stellar-sdk’s datastore takes
sep54-dirorsep54-s3; one that polls stellar-rpc takesrpc. Nothing else in the indexer changes: it is pointed at a loopback endpoint instead of the real one. - Write
describe. Name, target version, inputs, units (singlefirst;chainif the indexer can start at ledger 2 of a fresh network), the envelope (noneif there is no gate), and the observation kinds you can report.receiptsandtransactionsalone already catch drops, crashes, hangs and halts for the whole corpus. - Write
prepare. Create an empty store. If the indexer needs a starting cursor, set it torange.start − 1. - Write
ingest. Start the indexer on the given input and range, poll its progress marker, stop it when the marker reachesrange.end, and fill in the signals. - Write
observe. One query per kind, mapped onto the neutral names; pass stored values through asas_stored. - Write
teardown. Drop the store. - Run the self-check:
xdrcheck replay --adapter "./my-adapter" --target-version <v> --selftest(in Go, alsokittest.Checkin your own tests). - Replay:
xdrcheck replay --adapter "./my-adapter" --target-version <v> --input sep54-dir --report DIR.DIR/REPORT.mdstarts with the failures: for each, the entry and its CAPs, what the ledger holds against what the indexer held, and thexdrcheck replay --entries <id> …command that replays that entry alone. - Add kinds as you go. Each new kind switches on the expectations of that kind for every entry; kinds you
do not model stay
not_applicableand cost nothing.
observe.
8.1 wallet-backend’s declared scope
An indexer is judged against what it states it indexes, not against everything the ledger shows. The wallet-backend adapter (go/adapters/walletbackend/scope.go) declares, in the open out_of_scope property of
the participants, addresses and state_changes capabilities, the facts wallet-backend does not claim, each
rule with a basis that cites wallet-backend’s own source or documentation at the commit it was read at
(f0c00cf52420, the pinned current commit). It declares them at that commit and at any other commit where every
file the rules cite is byte-identical to the one read (so the cited statements and line numbers still hold, as at the
previous pins 3e22515a and 18db88f4); other commits, including every older pinned one, declare none.
The harness (
go/internal/match/scope.go) applies a rule to an expected fact when the fact meets every
criterion the rule names:
Such a fact is
not_applicable with “outside the adapter’s declared scope: <basis>” as its reason. A rule needs
a basis and at least one criterion of its kind; anything else makes the match fail. Negative facts
(must_not_include) and extras are never excused.
Facts wallet-backend does claim (for example a pool-share trustline removed by a revocation, a participant that
is the counterparty of a transfer, a contract deployed from an address) are not listed and fail when missing. A
statement of wallet-backend that the ledger contradicts is not a scope either: its code says a failed transaction
changes no state beyond its fee (internal/indexer/processors/participants.go:167), but a failed transaction
removes the pre-authorized transaction signers it carries (CAP-28), so a signer removal of a failed transaction
stays in scope and fails when it is missing.
Nor is behaviour reported upstream as a defect: wallet-backend writes ACCOUNT/CREATE for the create_account
operation only (internal/indexer/processors/token_transfer.go:289-298), so an account created by a native Stellar
Asset Contract transfer (CAP-73) gets no account_created. That defect is reported upstream and recorded as a known
divergence of the entries that show it, not declared as scope.
9. Conformance of an adapter
xdrcheck replay --selftest checks an adapter without scoring its indexer. It runs the adapter twice:
- Process A:
describe;prepare; aningestwith an input kind the adapter did not declare (must answerunsupported_input) and one with an undeclared unit (unsupported_unit); with a probe unit, a realingest,observeandteardown; then stdin is closed and the adapter must exit 0 within the grace period. - Process B:
describeagain, which must equal A’s; with a probe unit, a second session over the same unit, whose observation set must equal A’s.
timeout at the deadline, and no connection may leave loopback — and are reported as skipped by
--selftest. The harness’s own test, make harness-selftest, is different: it replays the corpus through the
seeded-bug indexers and checks that each lands in its documented class (replay-harness.md §12).