Migrate production contracts from multi_index/singleton to kv::table#291
Open
heifner wants to merge 2 commits intofeature/kv-scoped-tablefrom
Open
Migrate production contracts from multi_index/singleton to kv::table#291heifner wants to merge 2 commits intofeature/kv-scoped-tablefrom
heifner wants to merge 2 commits intofeature/kv-scoped-tablefrom
Conversation
Replaces multi_index/singleton with kv::table / kv::scoped_table /
kv::global across all six production contracts (sysio.token,
sysio.bios, sysio.msig, sysio.system, sysio.roa, sysio.authex).
Storage choices:
- kv::scoped_table — when keys naturally have a scope (sysio.token
accounts/stat, sysio.msig proposal/approvals,
sysio.roa nodeowners/policies/sponsors)
- kv::table — flat tables with no scope concept
(sysio.system finkeys/producers/blockinfo,
sysio.bios abihash, sysio.authex links)
- kv::global — single-row config / counter state
(sysio.system trxpglobal, sysio.roa fin_key_id,
producer payment globals)
Key wins:
- 18-38% WASM size reduction across the board (sysio.roa: -33%,
sysio.system: -18%, sysio.authex: -10%) from RAII handles, stack
buffers, and the if-constexpr scope path in kv::table.
- Drop-in replacement for kv_multi_index: kv::scoped_table produces
byte-identical primary keys, so no on-chain data migration needed.
- Per-table table_id namespace isolation (uint16 DJB2 hash) eliminates
the table-name byte cost in keys.
Supporting changes:
- libraries/testing/tester.cpp: get_row_by_id() now tries the scoped
key layout first ([scope:8B][pk:8B] for kv_multi_index/scoped_table),
then falls back to kv::global's bare [pk:8B] layout, so existing
helper APIs keep working across both storage shapes.
- plugins/producer_plugin/src/trx_priority_db.cpp: rewritten to read
trxpriority/trxpglobal under their new key formats (kv::global for
the singleton, kv::table for the priority list).
- programs/clio/main.cpp: msig review subcommand uses the correct
ABI field names in `find` payloads — `proposal_name`/`account`
instead of the legacy `primary_key`. Caught in code review of #290.
- contracts/sysio.system/include/sysio.system/sysio.system.hpp: imports
`sysio::kv::same_payer` instead of the now-removed `sysio::same_payer`.
- contracts/tests: finalizer_key, roa, and system_blockinfo tests
updated for the new storage layouts and bumped RAM where needed.
- test_contracts/blockinfo_tester recompiled against the new sysio.system
action set.
Reference data regenerated for the new on-chain action merkle roots:
- unittests/deep-mind/deep-mind.log
- unittests/snapshots/blocks.{index,log}, snap_v1.{bin.gz,bin.json.gz,json.gz}
- unittests/test-data/consensus_blockchain/{blocks.index,blocks.log,id,snapshot}
- tests/sysio_util_snapshot_info_test.py expected head_block_id
Depends on:
- #290 (unified get_table_rows API + the
secondary-index scope-prefix fix in fetch_primary)
- Wire-Network/wire-cdt#49 (kv::scoped_table, abigen secondary_indexes
for multi_index, canonical-RecordType checksum256 translation)
heifner
added a commit
to Wire-Network/hyperion-history-api
that referenced
this pull request
Apr 10, 2026
Wire-sysio PR #288 (table_id namespace isolation) replaced the legacy
key_format discriminator with a per-table table_id (uint16, DJB2 hash
of the table name % 65536) computed at compile time by CDT and emitted
on every contract_row_kv_v0 SHiP delta. The new payload shape is
{ code, payer, table_id, key, value } — no more scope, table, or
key_format fields.
Two breaks need fixing on the Hyperion side:
1. ABI binary format mismatch (definitions/abi_def.ts)
table_def gained table_id (uint16) and secondary_indexes
(vector<index_def>) in wire-sysio commit a9f4614f6, and `name` was
widened from sysio::name to a free-form string. Hyperion's
AbiDefinitions used by Serialize.getTypesFromAbi() to deserialize
contract ABIs from the SHiP `account` delta still had the old
layout, so any new wire-sysio ABI would either fail to parse or
misalign every field after the table name. Update table_def to
match the chain layout exactly and add an index_def struct.
2. KV delta handler logic (workers/deserializer.ts)
The contract_row_kv handler still gated on `payload.key_format !== 0`
(a field that no longer exists) and resolved the table by scanning
the ABI for entries with non-empty key_names/key_types — fragile,
and ambiguous now that every kv::table contract has multiple such
entries. Replaced with a direct table_id lookup:
abiObj.tables.find(t => t.table_id === payload.table_id).
For scoped tables (kv_multi_index, kv::scoped_table) the ABI
key_names start with "scope" — the handler now mirrors the decoded
scope value into payload.scope and the next field into
payload.primary_key, matching the legacy contract_row layout that
downstream handlers (notably *:accounts → storeAccount → ES
table-accounts index) read from. This keeps sysio.token end-to-end
identical to before:
1. SHiP emits contract_row_kv_v0 with table_id = hash("accounts")
2. Handler resolves kvTable.name = "accounts", decodes scope=alice
and sym_code=4607832 from the BE key, decodes the asset value
via abieos.hexToJson('sysio.token', 'account', value)
3. processTableDelta routes to *:accounts
4. *:accounts extracts @accounts.amount/symbol from data.balance
5. storeAccount indexes {code, scope, amount, symbol} to ES
6. /v2/state/get_tokens reads the same fields as before
Companion wire-sysio PRs:
- Wire-Network/wire-sysio#288 (table_id namespace isolation, SHiP
contract_row_kv_v0 / contract_index_kv_v0 emit table_id)
- Wire-Network/wire-sysio#290 (unified get_table_rows API — no
Hyperion HTTP API consumer is affected: get_currency_balance is a
dedicated endpoint and the only get_table_rows caller targets a
defunct `voters` table that no longer exists on Wire chains)
- Wire-Network/wire-sysio#291 (sysio.token migrated to
kv::scoped_table — primary keys are byte-identical to
kv_multi_index so on-chain data is unchanged)
Out of scope for this commit:
- master.ts:2079 getRows()/fillCurrentStateTables() — already dead
on Wire (no sysio.system voters table). Replacement / removal is a
follow-on cleanup.
- contract_index_kv handler — remains a no-op (secondary index data
is derivative).
- addons/wirejs-native cosmetic interface updates.
heifner
added a commit
to Wire-Network/wire-libraries-ts
that referenced
this pull request
Apr 10, 2026
Wire-sysio PR Wire-Network/wire-sysio#288 widened table_def.name from sysio::name (uint64) to a free-form std::string and appended two new fields — table_id (uint16, DJB2 hash of the table name % 65536) and secondary_indexes (vector<index_def>) — for KV-table per-table namespace isolation. The wire format break means any sdk-core caller that loads a contract ABI from on-chain via get_raw_abi (or any other binary-encoded source) parses misaligned starting at the first table field — name reads the wrong bytes, every subsequent field is shifted, and ricardian_clauses ends up reading garbage. PR Wire-Network/wire-sysio#290 separately changed the get_table_rows HTTP response shape for KV-backed tables. Each row used to be the decoded struct directly (or {data, payer} when show_payer was set); the unified endpoint now returns {key, value, payer?} per row. The ChainAPI wrapper used to destructure {data, payer} on show_payer and then map rows through Serializer.decode for typed callers — both paths break against the new shape. This commit takes a hard break on the binary format (no fallback to the legacy EOSIO 8-byte name encoding) and a backward-compatible unwrap on the HTTP response shape. packages/sdk-core/src/chain/Abi.ts: - new ABI.Index interface (name, key_type, table_id) mirroring sysio::chain::index_def in wire-sysio's libraries/chain/include/sysio/chain/abi_def.hpp. - ABI.Table gains optional table_id (uint16) and secondary_indexes (Index[]) fields. Optional so hand-built test fixtures and ABIs constructed programmatically don't need to specify them; the encoder defaults to 0 / [] when omitted. - fromABI() reads name as a length-prefixed string instead of an 8-byte sysio::name uint64, then consumes table_id (uint16 LE) and secondary_indexes (vector<index_def>) after the existing type field. Also reads a trailing protobuf_types extension after enums so that any subsequent extension appended to abi_def parses cleanly to EOF. - toABI() mirrors the parser: writes table_def.name via encoder.writeString(String(table.name)) (the String() coercion keeps NameType-typed callers compiling), then table_id and secondary_indexes, and finally the empty protobuf_types string. packages/sdk-core/src/api/v1/Chain.ts: - get_table_rows() now detects the unified KV row shape — each row has both `key` and `value` keys — and unwraps to the inner value before downstream processing. The old EOSIO shapes (decoded struct directly, or {data, payer} on show_payer) are left untouched, so sdk-core remains usable against EOSIO chains. - When show_payer is set on a KV row, payer is captured into ram_payers in the same way as the legacy {data, payer} path. Missing payer fields coerce to an empty Name rather than throwing. Tests (15 new cases, 304/304 sdk-core suite green): packages/sdk-core/tests/chain/Abi.test.ts (8 cases) - Round-trips a table with table_id + empty secondary_indexes - Round-trips a long table name (>12 chars) — the whole reason name was widened - Round-trips secondary_indexes with checksum256 key_type (the case the wire-cdt #49 abigen fix unblocked end-to-end) - table_id 0 and 65535 boundary values - Missing table_id defaults to 0 on encode + decode - protobuf_types extension is consumed without affecting enums - Encoder always emits the empty protobuf_types trailer - Multi-table ABI end-to-end round-trip (structs + actions + secondary_indexes) packages/sdk-core/tests/api/v1/Chain.test.ts (7 cases) - Unwraps the new {key, value} shape into plain rows - Unwraps the new shape with show_payer + captures ram_payers - Preserves legacy plain-row shape from EOSIO chains - Preserves legacy {data, payer} show_payer shape from EOSIO - Empty rows array works for both shapes - Missing payer in new shape coerces to empty name (no throw) - Uses an in-memory MockProvider so no network access required Out of scope: - packages/sdk-core/src/resources/{Ram,Rex,Powerup}.ts call get_table_rows for tables (rammarket, rex_pool, powup_state) that do not exist in wire-sysio and have not for some time. These resources were already dead on Wire chains regardless of any of the in-flight wire-sysio PRs and remain dead after this commit. - packages/sdk-core/src/types/SystemContractTypes.ts is auto- generated and would benefit from a regen pass after sysio.token's migration to kv::scoped_table merges (Wire-Network/wire-sysio#291), but the field schemas of the structs it generates do not depend on the binary table_def layout. Companion PRs: - Wire-Network/wire-sysio#288 — chain-side table_id namespace isolation - Wire-Network/wire-sysio#290 — unified get_table_rows API - Wire-Network/wire-sysio#291 — sysio.token migrated to kv::scoped_table - Wire-Network/wire-cdt#49 — kv::scoped_table + abigen secondary_indexes - Wire-Network/wirejs-native#4 — TS schema mirror for abi_def - Wire-Network/abieos#12 — abieos C API + binary format update - Wire-Network/node-abieos#8 — bumps abieos + drops the lossy uint64 round-trip - Wire-Network/hyperion-history-api#9 — KV delta handler + AbiDefinitions
heifner
added a commit
that referenced
this pull request
Apr 10, 2026
Wire's per-row KV value limit is 256 KiB and a direct setcode trx can carry ~512 KiB, so post-#291 sysio.msig setcode was capped well below what a non-msig setcode could deploy. This restores parity by chunking the serialized inner trx across rows of a new `propchunks` table when it exceeds an internal threshold (200 KiB), reassembling on `exec`, and exposing a read-only `getproposal` action that returns the assembled struct via /v1/chain/send_read_only_transaction. Storage layout - `proposal` row gains three appended `binary_extension` fields: `chunk_count`, `total_size`, `trx_hash`. Small proposals keep the legacy shape (full blob in `packed_transaction`, chunk_count = 0) so external tooling that reads the row directly via get_table_rows still works for the small case. - New `propchunks` `kv::scoped_table` keyed by (proposal_name, chunk_index) holds the bytes when a proposal is chunked; the parent row's `packed_transaction` is empty in that case. propose / approve / unapprove / cancel / exec - `propose` hashes the inner trx once via `sysio::sha256(trx_pos, size)` and stores it in `trx_hash`. Inner trxs at or below `proposal_chunk_size` go into `packed_transaction` as before; larger ones are split into ceil(size / proposal_chunk_size) `propchunks` rows. - Helpers `is_chunked`, `assemble_packed_trx`, `read_trx_header`, and `erase_proposal_chunks` keep the chunk-aware logic isolated. The header read pulls from chunk 0 alone for chunked proposals so approve doesn't pay reassembly cost just to inspect expiration / delay_sec. - `approve --proposal-hash` keeps the legacy `assert_sha256` path for inline proposals and uses the precomputed `trx_hash` for chunked proposals, since chunked rows have no inline blob to re-hash. - `exec` reassembles, deserializes, dispatches each action via `act.send()` exactly as before. Each individual dispatched action is still bounded by `max_inline_action_size`, so no chain-config change is required. - `cancel` and `exec` both call `erase_proposal_chunks` so chunk rows never outlive their parent proposal — no orphaned RAM. getproposal (read-only) - New `[[sysio::action("getproposal"), sysio::read_only]] proposal get_proposal(proposer, proposal_name)` returns the assembled proposal struct with `packed_transaction` always populated. CDT's codegen auto-emits `set_action_return_value` for non-void actions, and the chain skips `max_action_return_value_size` in read-only context, so the full ~512 KiB blob comes back via the trace's `action_traces[0].return_value`. clio multisig review now uses getproposal - Switched away from /v1/chain/get_table_rows on the `proposal` row. `multisig review` builds a one-action getproposal trx, posts it to /v1/chain/send_read_only_transaction, and pulls `processed.action_traces[0].return_value_data` (ABI-decoded by chain_plugin via the new action_results entry CDT generates from the return type). Storage-layout-agnostic — if sysio.msig refactors its tables again, clio keeps working as long as the getproposal signature is preserved. - This is the design principle for new clio commands going forward: read-only actions are the stable contract surface; get_table_rows reaches into table internals and is brittle (the same brittleness that bit us in #291 with ABI field-name matching). clio multisig propose_trx accepts structured action.data - Mirrors the try/catch fallback that `clio push transaction` already has at ~line 3145: try the literal `as<transaction>()` cast first, fall back to `abi_serializer::from_variant(trx_var, trx, abi_serializer_resolver, ...)` on the catch. Strict superset of the old behavior — old hex-data callers keep working unchanged, new callers can pass the natural JSON shape with structured `action.data` objects and clio will recursively encode via each contract's ABI. - Generic clio improvement that benefits anyone using propose_trx, not just the new msig integration test. Tests - contracts/tests/sysio.msig_tests.cpp adds three cases: - `big_transaction_chunked`: builds a >200 KiB inner trx (two setcode actions of sysio.system.wasm), proposes, exercises the chunked-path `--proposal-hash` check (correct hash succeeds, wrong hash rejected against the stored trx_hash), approves, and execs end-to-end. Default chain config — no setparams bumps. - `getproposal_read_only_returns_assembled`: pushes a read-only trx invoking getproposal, ABI-decodes the return value, asserts the reassembled `packed_transaction` byte-equals the original, and that `chunk_count > 0` and `total_size` matches. - `cancel_chunked_proposal_cleans_chunks`: proposes, cancels, re-proposes under the same name, approves, execs — would fail with "key already exists" on the second propose if cancel did not erase chunk rows. - New tests/multisig_review_test.py Python integration test launches a producer + non-producer API node cluster (with `--read-only-threads 1` on the API node), deploys the new sysio.msig wasm, submits a chunked propose via clio, runs `clio multisig review` against the API node so it actually goes through the read-only-trx getproposal path, and validates the JSON output: chunk_count > 0, total_size > 200 KiB, trx_hash populated, both setcode actions present, packed_transaction non-empty (proving the review fell through getproposal rather than echoing the empty inline row). Also runs `--show-approvals` which exercises the approvals2 / approvals / invals lookup paths. - All 117 contracts_unit_test cases, 14 read_only_trx_tests, and the full plugin_test suite pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrate all six production contracts from
multi_index/singletontokv::table/kv::scoped_table/kv::global. 18-38% WASM size reduction across the board (sysio.roa: -33%, sysio.system: -18%, sysio.authex: -10%) with no on-chain data migration required —kv::scoped_tableproduces byte-identical primary keys tokv_multi_index.Depends on:
get_table_rowsAPI and the secondary-index scope-prefix fix infetch_primarykv::scoped_table, abigensecondary_indexesformulti_index/kv_multi_index, canonical-RecordTypechecksum256translationStorage choices
kv::scoped_table<...>accounts/stat), sysio.msig (proposal/approvals/approvals2), sysio.roa (nodeowners/policies/sponsors/nodeownerreg/sponsorcount)kv_multi_indexkv::table<...>finkeys/producers/blockinfo/peerkeys/limitauthchg/trxpriority), sysio.bios (abihash), sysio.authex (links), sysio.system (finalizers)kv::global<...>Per-contract changes
accounts,statmigrated tokv::scoped_table. Usesupsert(payer, key, default, updater)for the optimal insert-or-modify path (1kv_get+ 1kv_set).abihashmigrated tokv::table.proposal/approvals/approvals2migrated tokv::scoped_table.invalsiskv::table(scope is alwayssysio.msigso flat is fine).finkeys/producers/blockinfo/peerkeys/limitauthchg/trxpriority/finalizersmigrated tokv::table. Trx-priority globals →kv::global. Importssysio::kv::same_payerinstead of the now-removedsysio::same_payer.nodeowners/policies/sponsors/nodeownerreg/sponsorcountmigrated tokv::scoped_table.reslimitiskv::table.fin_key_idgenerator →kv::global. Includes secondary indexes (bytier,bystatus,bytrxid checksum256).linksmigrated tokv::tablewith four secondary indexes includingbypubkey checksum256.Supporting changes
libraries/testing/tester.cpp—get_row_by_id()now tries the scoped key layout first ([scope:8B][pk:8B]forkv_multi_index/scoped_table), then falls back tokv::global's bare[pk:8B]layout, so existing helper APIs keep working across both storage shapes.plugins/producer_plugin/src/trx_priority_db.cpp— rewritten to readtrxpriority/trxpglobalunder their new key formats (kv::globalfor the singleton,kv::tablefor the priority list).programs/clio/main.cpp— msig review subcommand uses the correct ABI field names infindpayloads (proposal_name/accountinstead of the legacyprimary_key). Caught during code review of Unified get_table_rows API: scope, find, index_name #290.contracts/tests/—sysio.finalizer_key_tests,sysio.roa_tests,sysio.system_blockinfo_testsupdated for the new storage layouts and bumped RAM where needed.contracts/test_contracts/blockinfo_testerrecompiled against the newsysio.systemaction set.