Skip to main content
For indexer authors who write or debug an adapter: the program that lets xdrcheck replay drive one indexer. After this page you can implement every message, report every ingest outcome correctly, and pass --selftest. The message shapes are fixed by two JSON Schemas (2020-12): corpus/schema/adapter-protocol.schema.json and corpus/schema/observation.schema.json. The messages quoted below are in corpus/schema/examples/adapter-protocol/ and validate against them. Where this page and the schemas disagree, the schemas win. How the harness plans, matches and scores a run is in replay-harness.md.

1. What an adapter is

The harness knows nothing about any particular indexer. An adapter is a small program that knows one. On request it puts the indexer in a clean state, makes it ingest a range of ledgers from an input the harness serves, and reports what the indexer then holds, in a neutral vocabulary. An adapter never judges. It reports machine-observable facts; the harness compares them with the corpus manifest and does all scoring. Process signals alone cannot tell a bug halt from a deliberate stop on unsupported input, so the decision combines the adapter’s facts with what the manifest says about the entry.
Adapters shipped in this repository: go/adapters/walletbackend (the reference adapter), go/adapters/stellaretl, go/adapters/baseline (a minimal correct indexer and the complete example adapter; see its README), and go/adapters/minimal, the template for new adapters (§8).

2. Transport and framing

3. Versioning and capability negotiation

  • The protocol name carries a major version. Within xcc.adapter/1 changes are additive (new optional fields; new enum values only where the schema leaves a field open). Adapters ignore unknown request fields; the harness is strict about responses wherever the schema forbids additional properties, so typos are caught.
  • describe is always the first call. The harness lists the protocol versions it speaks; the adapter answers with one, or with the error unsupported_protocol_version.
  • Capabilities are declared, not discovered. describe states the inputs the indexer can read, the replay units the adapter can run, the observation kinds the indexer models (with subkinds and modelling choices), and the envelope: the protocols and meta versions the indexer at this version claims to handle. The harness plans the run from these declarations and records them in describe.json next to the report.

3.1 The envelope, and why “refused” is narrow

A deliberate fail-closed stop is safe and is not scored like a crash. A halt on valid input is a bug. From outside the process the two look the same, so the adapter declares its limits up front: Rules the harness enforces whatever the adapter says:
  1. A stop on an entry inside the envelope is a halt (mishandled · halt). An adapter cannot turn it into a refusal.
  2. refused is accepted only if the gate named in the response was declared in describe, the manifest confirms the entry really is outside that gate, the indexer wrote nothing for the refused ledger, and the process did not crash.
  3. Gates are a closed list, each with a predicate the harness checks against the manifest: min_protocol, max_protocol, min_protocol_with_transactions, ledger_close_meta_version, transaction_meta_version, operation_type.

4. Messages

Five methods, in this order within a session: describe (once per adapter process) → prepare → ingest → observe → teardown. A session is one replay unit: one fresh indexer state, one ledger range. Messages are shown wrapped where long; on the wire each is one line.

4.1 describe

timing sizes the harness’s ingest deadlines. participants.muxed is base_only (the indexer stores the base account) or base_and_alias (it also stores the muxed form beside it). address_encoding says how stored addresses are passed (strkey_text, version_byte_payload_hex, xdr_hex, raw_key_hex).

4.2 prepare

Puts the indexer at target_version into a clean state for one session. It may reuse anything in cachedir (built binaries, database templates) but nothing from another session’s data.
resolve_ranges exists because some targets have no release tags: the manifest’s known divergences then name commit ranges, and only the adapter holds the repository that decides whether the version under test lies inside one (A..B is every commit from A to B on B’s first-parent history, ..B everything up to B). For tagged packages the harness evaluates ranges itself and omits the field. (artifact_sha256 above is a placeholder.)

4.3 ingest

Makes the indexer ingest range from input and returns when it has finished the range, has stopped, or the deadline has passed.
Completed — wallet-backend at the commit that fixed wallet-backend#729:
Halted — the same ledger at the commit before that fix:
Refused — an indexer with a real gate (the baseline indexer built with a maximum protocol of 28) given the first Protocol 30 testnet ledger:

Input descriptor

range is inclusive and names real ledger sequences as served. network.passphrase is the passphrase of every ledger in the range: one harness pass per passphrase, because readers verify transaction hashes against it. schema mirrors the store’s .config.json (ledgersPerBatch, batchesPerPartition). The dummy credentials matter: without them an AWS client probes the instance-metadata address, which is an outbound request.

Ingest statuses and the facts that justify each

The adapter chooses the first row whose facts hold. stderr_tail is for the report only; no status may rest on matching log text. last_ledger is always the indexer’s own account of how far it got. rows_for_failing_ledger lets the harness see partial writes.

4.4 observe

Returns what the indexer holds for a range as an observation set (xcc.observation/v1). It is called after ingest whatever the status, so partial progress is visible.
That is wallet-backend at the commit that fixed wallet-backend#716. At the commit before it, the same call returns the signer as 7844ad…879b20 — 33 bytes, the signed payload dropped — and nothing else differs. The harness decodes the stored bytes itself and finds they are not the expected P… key.

Observation rules

  • Report what is stored, exactly. Every address-bearing value has an as_stored form: the literal text or bytes the indexer keeps. Interpreted fields (address, signer.key, on_chain_form) are filled only when the indexer itself holds that form. An adapter never computes a base account from a muxed one, re-derives a strkey, pads, trims or rounds on the indexer’s behalf: that would hide exactly the bugs under test.
  • Amounts and 64-bit IDs are decimal strings. A value that is not an exact integer in the indexer (a JSON float, a decimal with a fraction) goes literally into amount_as_stored; amount is left out.
  • One stored record is one fact. An adapter maps each record to at most one fact of the closest neutral kind, never merging or splitting records or filling a field the record lacks. Differences between indexers’ models are absorbed by the declarations in describe (subkinds, modelling, muxed), not by the adapter. An indexer that writes one balance row per event (per mint, transfer or path hop) is reported row by row; the harness compares those rows with the reference’s net change by their sum (manifest.md §6.6, S-7).
  • State-change detail keys are closed per kind, the same list as the manifest’s (manifest.md §6.6), enforced by the observation schema. A value the indexer holds in another representation goes under <key>_as_stored as the literal token, which is evidence only.
  • Result codes use XDR-JSON names (tx_success, tx_failed, …) mapped from the indexer’s labels through a fixed table; the original may be kept in native_label.
  • Kinds are optional; absence is declared. Each kind is provided or not_applicable with a reason. An expected fact whose sub-kind is not listed is not_applicable for that fact. A provided kind that comes back empty means the indexer holds nothing — that is how drops are seen. A ledger missing from ledgers was not seen.
  • Locators. Facts carry tx_hash wherever the indexer can supply one, and tx_index / op_index when it has them. Facts without a transaction locator are matched at ledger scope.

4.5 teardown

Stops the indexer if it is still running and removes the session’s state (databases, buckets, temporary files). With keep_state: true (xdrcheck replay --keep-state) the adapter leaves everything in place and lists it in left_behind.

4.6 Errors

An error response means the adapter or its environment failed; it is never a statement about how the indexer handled a ledger. The harness records the affected entries as harness_fault.

5. Determinism

  • No wall-clock values in results except wall_ms, a duration used for reporting and deadline sizing, never for scoring.
  • Arrays in an observation set are ordered: ledgers by ledger_seq; within a ledger, facts by transaction, then operation, then the fact’s JSON text. Two runs on the same input give byte-identical observation sets once wall_ms is removed.
  • The adapter keeps the system clock out of observations: where the indexer stamps rows (ingested_at), that column is left out.
  • Session IDs, ports and directories come from the harness; the adapter invents none.

6. Resource limits — owned by the harness

  • Time. Deadlines per call:
  • Response size. xdrcheck replay refuses a response line longer than 64 MiB (a harness_fault for that unit) and asks for observations in windows of consecutive ledgers, so an adapter never returns a long unit in one line.
  • Memory and processes. The harness starts the adapter as the leader of a new process group inside its own cgroup v2 scope (MemoryMax, swap off, TasksMax; xdrcheck replay --memory-max, --pids-max), so everything the adapter spawns is inside it. Where no user cgroup is available it falls back to an address-space limit plus polling of resident memory, and the report says which mechanism was used. An out-of-memory kill of the indexer surfaces as crashed; one of the adapter itself is recorded by the harness as mishandled · crash.
  • Services an adapter runs in containers are outside that group. The adapter receives the limits in prepare.limits and passes them on.

7. Security rules

  1. No network. Adapter and indexer must work with outbound networking disabled; make verify-replay OFFLINE=1 runs the whole harness in a network namespace with loopback only. Every URL in a request is a loopback address (the schema enforces it). The adapter sets every endpoint of the indexer explicitly, because defaults may point at public infrastructure.
  2. No credentials. The protocol carries no secrets; object-store credentials are dummy values for the local endpoint. The indexer is started with a scrubbed environment.
  3. No production data. An adapter works on a fresh store it creates in the session.
  4. No disclosure required. Nothing in the protocol needs the indexer’s source, schema or internal names; field and native_label are optional, so an adapter for a private indexer can leave them out. xdrcheck report redact strips evidence strings from a report before it is shared.
  5. Corpus bytes are untrusted input from the indexer’s point of view (some entries are built to stress readers). Adapters do not pre-validate or filter them.
  6. Telemetry in dependencies is switched off (the wallet-backend adapter disables TimescaleDB’s telemetry job).

8. Writing an adapter

Start from a template:
  • Go: go/adapters/minimal — about 150 lines, receipts and transactions, every place to change marked REPLACE. It is built on the public package go/adapterkit (github.com/OWNER/REPO/go/adapterkit, standard library only): the message types, Main/Serve (the stdio transport, the version check, error responses, the refusal of undeclared inputs and units), RunChild (an indexer process with a deadline), ProvidedKinds and SortFacts, the embedded schemas with the harness’s own response check, and kittest, the selftest’s checks in process for your own go test. Copy main.go and protocol_test.go into your own module; nothing under internal/ is needed (a test builds the copy as a module of its own).
  • Any other language: examples/adapter-ts is the same template in TypeScript. The contract is only the two schemas, corpus/schema/adapter-protocol.schema.json and observation.schema.json.
  • Every kind: go/adapters/baseline is the complete example: an indexer that is a separate process with its own store, and all six kinds.
scripts/demo.sh runs the Go template with one planted bug and prints what the report says about it. For any indexer:
  1. Choose the input. An indexer that reads a data lake through go-stellar-sdk’s datastore takes sep54-dir or sep54-s3; one that polls stellar-rpc takes rpc. Nothing else in the indexer changes: it is pointed at a loopback endpoint instead of the real one.
  2. Write describe. Name, target version, inputs, units (single first; chain if the indexer can start at ledger 2 of a fresh network), the envelope (none if there is no gate), and the observation kinds you can report. receipts and transactions alone already catch drops, crashes, hangs and halts for the whole corpus.
  3. Write prepare. Create an empty store. If the indexer needs a starting cursor, set it to range.start − 1.
  4. Write ingest. Start the indexer on the given input and range, poll its progress marker, stop it when the marker reaches range.end, and fill in the signals.
  5. Write observe. One query per kind, mapped onto the neutral names; pass stored values through as as_stored.
  6. Write teardown. Drop the store.
  7. Run the self-check: xdrcheck replay --adapter "./my-adapter" --target-version <v> --selftest (in Go, also kittest.Check in your own tests).
  8. Replay: xdrcheck replay --adapter "./my-adapter" --target-version <v> --input sep54-dir --report DIR. DIR/REPORT.md starts with the failures: for each, the entry and its CAPs, what the ledger holds against what the indexer held, and the xdrcheck replay --entries <id> … command that replays that entry alone.
  9. Add kinds as you go. Each new kind switches on the expectations of that kind for every entry; kinds you do not model stay not_applicable and cost nothing.
An indexer with no query interface can still be adapted: have it write its records somewhere in your own format and translate them in observe.

8.1 wallet-backend’s declared scope

An indexer is judged against what it states it indexes, not against everything the ledger shows. The wallet-backend adapter (go/adapters/walletbackend/scope.go) declares, in the open out_of_scope property of the participants, addresses and state_changes capabilities, the facts wallet-backend does not claim, each rule with a basis that cites wallet-backend’s own source or documentation at the commit it was read at (f0c00cf52420, the pinned current commit). It declares them at that commit and at any other commit where every file the rules cite is byte-identical to the one read (so the cited statements and line numbers still hold, as at the previous pins 3e22515a and 18db88f4); other commits, including every older pinned one, declare none. The harness (go/internal/match/scope.go) applies a rule to an expected fact when the fact meets every criterion the rule names: Such a fact is not_applicable with “outside the adapter’s declared scope: <basis>” as its reason. A rule needs a basis and at least one criterion of its kind; anything else makes the match fail. Negative facts (must_not_include) and extras are never excused. Facts wallet-backend does claim (for example a pool-share trustline removed by a revocation, a participant that is the counterparty of a transfer, a contract deployed from an address) are not listed and fail when missing. A statement of wallet-backend that the ledger contradicts is not a scope either: its code says a failed transaction changes no state beyond its fee (internal/indexer/processors/participants.go:167), but a failed transaction removes the pre-authorized transaction signers it carries (CAP-28), so a signer removal of a failed transaction stays in scope and fails when it is missing. Nor is behaviour reported upstream as a defect: wallet-backend writes ACCOUNT/CREATE for the create_account operation only (internal/indexer/processors/token_transfer.go:289-298), so an account created by a native Stellar Asset Contract transfer (CAP-73) gets no account_created. That defect is reported upstream and recorded as a known divergence of the entries that show it, not declared as scope.

9. Conformance of an adapter

xdrcheck replay --selftest checks an adapter without scoring its indexer. It runs the adapter twice:
  • Process A: describe; prepare; an ingest with an input kind the adapter did not declare (must answer unsupported_input) and one with an undeclared unit (unsupported_unit); with a probe unit, a real ingest, observe and teardown; then stdin is closed and the adapter must exit 0 within the grace period.
  • Process B: describe again, which must equal A’s; with a probe unit, a second session over the same unit, whose observation set must equal A’s.
Every response is validated against the schemas, so a broken message ends the self-check with the protocol error’s reason. Two checks need the ledger server or a network namespace — a ledger deliberately withheld must end in timeout at the deadline, and no connection may leave loopback — and are reported as skipped by --selftest. The harness’s own test, make harness-selftest, is different: it replays the corpus through the seeded-bug indexers and checks that each lands in its documented class (replay-harness.md §12).