Prerequisites
-
Linux on x86-64 with cgroup v2 and unprivileged user namespaces (for
OFFLINE=1), Docker, and about 15 GB of disk. See the requirements. -
The full fetch, which includes stellar-core, its goldens and every dependency:
Steps
1
Register the ID
An entry ID is born in
corpus/plan.json and nowhere else. Search the plan first: the pattern may already exist
under another name. If it is new, add a row with every key the plan schema (corpus/schema/plan.schema.json)
requires: kind, CAPs, minimum protocol, source, priority, what it serves. Run make repo-checks, and send the
row as its own change.One entry is one pattern. The same pattern on another network or protocol is a variant of the same entry.2
Produce the bytes
Pick the source in this order, and never edit bytes:
- Local stellar-core
- stellar-core golden
- Network history
- Synthetic
Write a scenario under The generator runs a standalone, offline stellar-core with manual close. Anyone can reproduce the bytes.
Scenario placeholders (
corpus/scenarios/ (schema corpus/schema/scenario.schema.json) and run it on
both passphrases:${acct.L}, ${muxed(L, id)}, ${signer.payload(L, hex)}, …) and setup items are
listed in Writing a scenario.3
Render the decoded value
requires, derived.l1).4
Write expect
State what a correct reader must derive, from two things only: the scenario’s intent and the decoded bytes.
Never take an expected value from the output of an SDK or indexer under test.
Set
validity right after synthetic: valid for anything stellar-core wrote. See Entries.5
Compare with the reference extractor
6
Index and verify
verify-corpus checks that the entry round-trips through rs-stellar-xdr, that its sidecar and pinpoints agree,
that the reference extractor agrees with expect, and that golden entries equal stellar-core’s goldens.
verify-coverage checks that the coverage report lists the new entry under its CAP.What success looks like
Every command above exits 0, and the entry appears under its CAP incorpus/COVERAGE.md. Before you send the
change, answer each question of the checklist for every entry.
Troubleshooting
verify says a value cannot be derived
verify says a value cannot be derived
A value you stated that the extractor cannot derive from the bytes fails
verify. Correct it, or keep it with
"unverified": "<reason>" on the item.gen --check reports different bytes
gen --check reports different bytes
Generation must be reproducible: the same ledger header and transaction hashes, and the same bytes after
stellar-core’s golden normalisation. A scenario that depends on wall-clock time or unordered input breaks this.
A protocol is above what the build supports
A protocol is above what the build supports
The oracle refuses a ledger above
max_protocol. Shapes of the next protocol go into the -next channel; see
Maintain the kit through a protocol upgrade.Next steps
- Authoring reference, below: the full workflow, scenario syntax and worked examples.
- Manifest and index format: every field.
- CAP coverage: every coverage row and its entries.
- CLI reference: every
xdrcheck-corpuscommand.
Authoring reference
How to add or change an entry so that it passesverify. The normative rules are in
manifest.md (fields and matching) and entry-ids.md (IDs); this page is the
workflow, the scenario syntax and the checklist to answer before you send an entry.
1. Before you start
- The ID must already be a row in
corpus/plan.json. If you need a new entry, add the plan row first, in its own change, asentry-ids.mdsection 5 describes (kind, CAP, minimum protocol, source, priority, what it serves). One entry is one pattern; the same pattern on another network or protocol is a variant of the same entry, not a new entry. - Pick the source in this order: a local stellar-core scenario (reproducible by anyone, on both passphrases);
a stellar-core golden at the pinned tag; an unmodified public-lake object; and only when stellar-core cannot or
would not emit the bytes, a synthetic recipe (
synthetic: true, with the reason). The five source kinds and what each must record are inmanifest.mdsection 4.4. - Never edit bytes. A blob is exactly what stellar-core emitted or the lake served. A smaller value cut out of
a ledger is an
xdrentry withslice_ofpointing to its parent, not an edit. A synthetic ledger is anxdrentry of typeLedgerCloseMetawhose recipe edits a local-core entry’s blob (the hashes the edit invalidates,scpValue.txSetHashand the header hash, recomputed);synthwrites its variant with networknone, so set the base ledger’snetworkandpassphraseon it, which the reference extractor needs to hash the envelopes.
2. Writing expect
expect states what a correct reader must derive. Write it from two things only:
- the scenario’s intent — what the transaction was built to do (for local-core entries the scenario step says so);
- the decoded bytes — the SEP-51 sidecar the oracle emits, read by hand.
xdrcheck-corpus extract <entry>, manifest.md section 6.11) derive
participants, state changes and movements from ledger-entry changes and consensus events, and resolve every
disagreement by reading the bytes again. Do not resolve a disagreement by copying either side.
Passphrase-dependent values (transaction hashes, contract IDs, fees that differ per protocol) go under each
variant’s
derived and are referenced as ${derived.<name>} (manifest.md section 4.6).
A value you state that the extractor cannot derive from the bytes (a modelling detail no ledger field carries) fails
verify. Correct it, or keep it with "unverified": "<reason>" on the item (manifest.md sections
6.4 and 6.11). Two variant fields are derived, not written: requires (XDR features the bytes need; mandatory on
every variant, [] when none) and derived.l1 (what an SDK’s object layer reads from the focus, section 10.1).
xdrcheck-corpus render <entry> fills both (with the sidecar fields), synth, slice and gen fill them for the
entries they write, and xdrcheck-corpus verify prints the exact JSON it expects. A negative variant has no sidecar:
render sets its requires to [] unless you state the feature its defect needs.
A variant whose SEP-51 JSON does not round-trip declares "json_roundtrip": {"skip": true, "reason": "…"} (section 4.2).
Every entry states its validity (manifest.md section 9.5), right after synthetic (or
negative):
Add
core-lenient to flags when core_behaviour.accepts is true. The schema and verify reject a validity
that disagrees with negative, the reject reason or the api block, so pick the row from what the entry already
says. xdrcheck-corpus gen, intake and synth write a starting value (valid, or malformed for a negative plan
row); confirm it. After adding or changing entries, run xdrcheck-corpus index to rewrite corpus/index.json.
3. Known divergences
A known divergence records that a specific reader gets this entry wrong in a version range (manifest.md section 7). It must cite an executed result (a transcript, a report, a
reproduction under evidence/), the target and the version range or commit range, the level (L0–L4 or
replay) and the outcome; for replay also the qualifier (halt, wrong_value, partial, …). A divergence known
only from source reading or an upstream report is recorded with observed: source-reading or reported, and is
asserted only after it has been executed once. A divergence that is security-relevant and not yet disclosed does not
go in the public manifest.
4. Worked examples
The three entries vendored first are the reference for style:5. Writing a scenario (local-core entries)
A local-core entry comes from a scenario file undercorpus/scenarios/, validated against
corpus/schema/scenario.schema.json and run with xdrcheck-corpus gen <file> (then xdrcheck-corpus gen --check <file>).
Transactions are SEP-51 JSON of Transaction with ${…} placeholders:
${signer.pre_auth_tx(STEP)} may name a later step (add the signer, then submit the pre-authorised transaction with
sign: [{pre_auth: true}]); that step must not be simulate: true, and its transaction must not depend on another
pre-auth hash. A hash-x signature is sign: [{hash_x: "${hash_x.preimage(NAME)}"}] (or the preimage in hex).
Write cond: none (PRECOND_NONE) unless the entry is about preconditions or the step needs a time bound (a
transaction core must drop, an ${order_nonce}). Unbounded time bounds ({time: {min_time: "0", max_time: "0"}})
change no outcome, but they are part of the envelope, which the golden mirror (docs/manifest.md section 4.7) compares unmasked, and almost no transaction in stellar-core’s golden set carries them: a focus
transaction with them finds no golden counterpart.
Setup items: fund: {L: "<XLM>", …}; trust: {L: ASSET | [ASSET, …], …} (one transaction, maximum limits);
pay: {from: L, to: L, asset: ASSET | native, amount: "<decimal>"}; deploy_sac: ASSET; deploy_contract: NAME;
tx: <template>; close: {ledgers: n}; config_upgrade: {…} (section 5.2). Any item may name an entry (section 5.1).
A step’s upgrade: {…} arms ledger upgrades for the ledger the step closes, with the parameter names of
stellar-core’s upgrades command: basefee, basereserve, maxtxsetsize, maxsorobantxsetsize (Protocol 20 and
later), flags, protocolversion. A step may hold an upgrade and no transaction. Upgrades are applied after the
ledger’s transactions, so the header of that ledger already shows the new value while its transactions were applied
under the old one.
Soroban:
deploy_sacdeploys the asset’s contract from root in one ledger, at any protocol.contracts.NAMEgiveswasm(relative to the corpus root, for examplecontracts/shape-emitter.wasm),wasm_sha256,deployer(an actor; root when absent) andconstructor_args(SEP-51ScValJSON, for example[{symbol: xcc}]). The salt is sha256 ofxcc/contract-salt/<NAME>, so${contract.id(NAME)}is known before the run when a deployer is named.deploy_contractuploads the code (one ledger, skipped when it is already on the ledger) and creates the contract (one ledger) through simulation, so a scenario withdeploy_contractlists only Protocols 29 and later.- A step with a declared footprint writes
ext.v1itself (soroban_resources: declared). A step withsimulate: true(Protocol 29 or later; limit it withprotocolsin a scenario that also runs at 28) writesext: v0or leavesextout, may leave the operation’sauthout, and gives the inclusion fee asfee: the generator simulates the oneinvoke_host_functionwith the pinned host against the node’s state, fills the footprint, resources, resource fee and recorded authorization (source-account credentials when the authorizer is the transaction source; anything else needsauthwritten out), and records the step as simulated (source.soroban_resources: simulated). The applied transaction must match its simulation (success, contract events, return value).
derived.accounts (actor label → G…), so an expect shared with a variant
from another source (a stellar-core golden, whose accounts differ) names an actor as ${derived.accounts.<label>}
and the other variant gives the same labels in its own derived.
YAML nests to a fixed limit of the parser and each level of an ScVal vector or an authorization
tree is two of them. A scenario whose values nest deeper (a near-limit argument or authorization tree) is written
as a .json file, which has no nesting limit.
A step’s mirrors has exactly the manifest entry’s mirrors shape (docs/manifest.md, the golden mirror); gen
copies it into the entry.
5.1 Entries on bring-up and setup ledgers; the root account
Every chain starts with the bring-up: ledger 2 upgrades the fresh node from genesis (protocol 0) to the target in one jump (itsLedgerCloseMeta is v0, and the upgrade creates the configuration settings of every protocol
crossed), then two Soroban settings sets (enable, then the protocol’s network limits) are written and applied.
A scenario names entries on these ledgers with bring_up, and on a setup item’s ledger with entry next to the
item’s action. Both take expect, fields and notes like a step:
source.step is bring-up-<ledger> or setup-<index>-<action>; a setup item’s transaction is the
variant’s focus.
The label root names the network’s root account (its seed is the network ID, so it differs per passphrase):
${acct.root}, ${seq.root}, ${muxed(root, id)}, sign: [{key: root}] and fee_bump.fee_source: root. In
expect, fields and mirrors, ${acct.root} is written as ${derived.accounts.root}, which gen fills in
every variant of a scenario that names root. No actor may be called root.
Protocols. A scenario may list any protocol from 13 (the first that accepts ENVELOPE_TYPE_TX envelopes) to 30
except 20. Below 20 the bring-up is the upgrade ledger alone: there is no Soroban and no settings set, so steps start
at ledger 3 and the chain index lists no settings files; LedgerCloseMeta is v0 throughout. From 21 the bring-up
applies enable and the protocol’s network set, vendored under corpus/settings/ from quickstart’s
config-settings/p<N>/testnet.json at the pinned commit (28 borrows 27’s; quickstart has no file for 20, so 20 is
refused). LedgerCloseMeta is v1 at 20–22 and v2 from 23. A stepwise upgrade (for example 23 → 24, which changes the
fee pool on the public passphrase) is a step upgrade: {protocolversion: 24} with an entry, limited with
protocols to the chain it upgrades.
5.2 Changing Soroban settings: config_upgrade
A setup item config_upgrade changes Soroban configuration settings the way the bring-up applies its sets: root
writes a ConfigUpgradeSet through its own upgrade contract (restore, upload, create, invoke; one ledger each) and
the next ledger applies it, so the item takes five ledgers. The contract’s salt is sha256 of
xcc/settings-upgrade-salt/<scenario>/setup-<index>, so the chain reproduces. Keys are ConfigSettingEntry arm
names; an object is merged into the setting the node holds (only the fields written change), anything else — and
any setting the node does not hold, such as the *_delta arms — is the arm’s SEP-51 JSON as written. Frozen keys
may be written as LedgerKey JSON instead of XDR hex. Protocol 20 or later; core refuses a set that fails its
validity checks (for example min_persistent_ttl below 10) and the run fails.
source.inputs: <scenario>/setup-<index>.xdr with its sha256).
5.3 A transaction left out of its ledger: expect_excluded
Surge pricing leaves the lowest bids out of a ledger when more operations are queued than it admits. A step with
expect_excluded: true (and no expect_result) must be accepted at submission and absent from the ledger its group
closes; it consumes no sequence number and is never the entry’s focus. Core keeps it queued, so give it time bounds
that end with that ledger, and lower the ledger’s limit first with an upgrade step:
base_fee (the lowest included bid, 200 here) that every included transaction is
charged. Under surge pricing the order model’s default guess of the set (the ledger’s base fee) is wrong; gen
learns the real shape from a first attempt (section 5.4).
5.4 Several transactions in one ledger: order
stellar-core applies a ledger’s transactions in an order derived from the transaction-set hash: the classic phase
in per-account batches, each batch sorted by transaction hash XOR set hash; the Soroban phase (Protocol 23 and
later) cluster by cluster in the same key. The order therefore differs per passphrase and changes with every byte.
To fix it, the first step of the ledger group (the one the others name in same_ledger_as) lists the group’s
transaction steps in the order wanted, and one template of the group uses ${order_nonce}:
gen builds the group for ${order_nonce} = 0, 1, 2, … until its model of core’s transaction set (the set’s
structure, every container sorted by full hash, lessThanXored apply sorting) gives that order, per network and
protocol, and records the nonce and the shape it used in the chain index (orders, also in run.json). After the close it checks the order core
really applied. If the model’s guess of the set’s structure (one component at the base fee; all Soroban
transactions in one cluster) was wrong, it reads the structure core built from that ledger and runs the scenario
again from a fresh node. The search is deterministic; gen --check repeats it and reports a nonce or shape other
than the chain records as a difference. Transactions of one
source account always apply in sequence-number order, and classic ones before Soroban ones; an order that
contradicts this fails after 20,000 nonces. Put the nonce in a fixed-width field (a memo id, a time bound): twins
must have equal blob lengths. With order, expect.transactions[].index is the same in every variant. A memo id
nonce also becomes the payment’s routing ID in classic events, which then differs per variant; a time bound does
not. gen refuses two transactions of one ledger with the same fee: give each a distinct fee (another operation
raises the fee without changing what it bids per operation).
6. Checklist before you send an entry
Answer each question for every entry, as someone who has not seen the work would; any “no” means the entry is not ready.- Do the blob’s bytes match the provenance (lake object and key, golden file and frame, or scenario run)?
- Is every expectation traceable to the scenario’s intent or to a field in the decoded sidecar — and not to the output of an SDK or indexer under test?
- Does the reference extractor agree with
participants,state_changesandmovements? Where it does not, is the reason written down? - Are the must-not facts present where the pattern is a trap?
- Does every known divergence cite an executed result, or say honestly that it does not?
- Are the field-table citations (
fields) the right rows, under the right status? - Is the entry one pattern, with variants rather than copies for other networks or protocols?
- Is
validityright for what a correct reader does with the input, and do theflagsname its bug class?
xdrcheck-corpus index and make repo-checks: the checks validate the entries against
the schema, check every blob, sidecar and captured object hash, every cited repository path, every entry ID, and
that corpus/index.json matches the manifest.