Unified get_table_rows API: scope, find, index_name#290
Open
heifner wants to merge 5 commits intofeature/kv-table-idfrom
Open
Unified get_table_rows API: scope, find, index_name#290heifner wants to merge 5 commits intofeature/kv-table-idfrom
heifner wants to merge 5 commits intofeature/kv-table-idfrom
Conversation
Replaces both old get_table_rows and get_kv_rows with a single unified endpoint backed by the KV implementation. Fixes scope type ambiguity (AntelopeIO/spring#1379) and get_table_by_scope pagination loops (AntelopeIO/spring#615). API changes: - Remove /v1/chain/get_kv_rows endpoint (merged into get_table_rows) - Add optional `scope` field — parsed via ABI key_types[0] with uint64 fallback (name/uint64/symbol all work, including scope=0) - Add `find` field — exact key lookup, errors if combined with bounds - Add `index_name` — accepts named indexes or numeric positions ("2") - Response format: rows are {key, value, payer?} objects - Default json=true (was false), limit=50 (was 10) - Dropped fields (key_type, encode_type, index_position) silently ignored Clio changes: - `get table <code> <table>` — 2 positional args (was 3) - `--scope`, `--find`, `--index` flags replace positional scope - msig subcommand updated to use `find` and new row format Test coverage (18 C++ test cases): - Scoped queries, scope isolation, scope type from ABI (#1379) - find exact lookup, find+bounds error validation - index_name named and numeric, invalid index error - Pagination with scope-stripped next_key - show_payer, reverse, bounds, hex mode - get_table_by_scope pagination no-loop (#615) Also updates Python integration tests and docs.
The unified get_table_rows endpoint's secondary index path silently
returned empty `value` strings for any scoped multi_index /
kv_multi_index / kv::scoped_table because fetch_primary looked up the
primary row using only the secondary's stored pri_key reference (the
in-scope portion). For scoped tables the actual primary key in the KV
store is [scope:8B BE][pri_key], so the find missed and the value
field came back blank. Decoded keys came back as 8-byte hex blobs for
the same reason.
Fix: in fetch_primary, build the full primary key by prepending
scope_prefix_bytes (already computed in the enclosing function) and
use that for both the pri_idx find and the row's r.key field. For
unscoped kv::table, scope_prefix_bytes is empty and the behavior is
unchanged.
Test coverage additions in tests/get_table_tests.cpp:
get_table_next_key_test (extended with sec-1..sec-9):
* sec-1..sec-3 — multi_index secondary by name (uint64), numeric
position, and uint128 key type
* sec-4 — multi_index secondary with checksum256 key (the canonical
RecordType abigen path that was just fixed in wire-cdt #49)
* sec-5 — invalid index_name on multi_index → throws
* sec-6 — secondary pagination with limit + next_key
* sec-7 — secondary reverse ordering
* sec-8 — secondary hex bounds, using kv_scoped_key_size /
kv_scope_prefix_size constants for [scope:8B][sec:8B] layout
* sec-9 — secondary with show_payer
get_table_find_scoped_test (new):
* find on scoped table without scope → contract_table_query_exception
('Cannot use find on a scoped table without specifying scope')
* find on scoped table with scope → exact match
* find on scoped table with scope but missing key → 0 rows
get_table_missing_contract_test (new):
* get_table_rows on account with no contract → throws
* get_table_rows on non-existent table name → throws with
'not specified in the ABI'
* get_table_by_scope on account with no contract → 0 rows, no exception
* get_table_by_scope on contract with no data → 0 rows, no exception
The new sec-1..sec-9 cases require the get_table_test contract's ABI
to declare secondary_indexes for its multi_index tables. Recompiling
the test contract with the wire-cdt #49 fix produces an ABI where
hashobjs.bysec1/bysec2 correctly emit checksum256 (was fixed_bytes<32>
before that fix).
Also adds a "Test Binaries" section to CLAUDE.md documenting that
unit_test does NOT contain everything: tests in tests/get_table_tests.cpp
and tests/test_*.cpp build into plugin_test, and the standard pre-PR
sweep should run unit_test, plugin_test, contracts_unit_test, and
test_fc.
heifner
added a commit
that referenced
this pull request
Apr 10, 2026
Replaces multi_index/singleton with kv::table / kv::scoped_table /
kv::global across all six production contracts (sysio.token,
sysio.bios, sysio.msig, sysio.system, sysio.roa, sysio.authex).
Storage choices:
- kv::scoped_table — when keys naturally have a scope (sysio.token
accounts/stat, sysio.msig proposal/approvals,
sysio.roa nodeowners/policies/sponsors)
- kv::table — flat tables with no scope concept
(sysio.system finkeys/producers/blockinfo,
sysio.bios abihash, sysio.authex links)
- kv::global — single-row config / counter state
(sysio.system trxpglobal, sysio.roa fin_key_id,
producer payment globals)
Key wins:
- 18-38% WASM size reduction across the board (sysio.roa: -33%,
sysio.system: -18%, sysio.authex: -10%) from RAII handles, stack
buffers, and the if-constexpr scope path in kv::table.
- Drop-in replacement for kv_multi_index: kv::scoped_table produces
byte-identical primary keys, so no on-chain data migration needed.
- Per-table table_id namespace isolation (uint16 DJB2 hash) eliminates
the table-name byte cost in keys.
Supporting changes:
- libraries/testing/tester.cpp: get_row_by_id() now tries the scoped
key layout first ([scope:8B][pk:8B] for kv_multi_index/scoped_table),
then falls back to kv::global's bare [pk:8B] layout, so existing
helper APIs keep working across both storage shapes.
- plugins/producer_plugin/src/trx_priority_db.cpp: rewritten to read
trxpriority/trxpglobal under their new key formats (kv::global for
the singleton, kv::table for the priority list).
- programs/clio/main.cpp: msig review subcommand uses the correct
ABI field names in `find` payloads — `proposal_name`/`account`
instead of the legacy `primary_key`. Caught in code review of #290.
- contracts/sysio.system/include/sysio.system/sysio.system.hpp: imports
`sysio::kv::same_payer` instead of the now-removed `sysio::same_payer`.
- contracts/tests: finalizer_key, roa, and system_blockinfo tests
updated for the new storage layouts and bumped RAM where needed.
- test_contracts/blockinfo_tester recompiled against the new sysio.system
action set.
Reference data regenerated for the new on-chain action merkle roots:
- unittests/deep-mind/deep-mind.log
- unittests/snapshots/blocks.{index,log}, snap_v1.{bin.gz,bin.json.gz,json.gz}
- unittests/test-data/consensus_blockchain/{blocks.index,blocks.log,id,snapshot}
- tests/sysio_util_snapshot_info_test.py expected head_block_id
Depends on:
- #290 (unified get_table_rows API + the
secondary-index scope-prefix fix in fetch_primary)
- Wire-Network/wire-cdt#49 (kv::scoped_table, abigen secondary_indexes
for multi_index, canonical-RecordType checksum256 translation)
heifner
added a commit
to Wire-Network/hyperion-history-api
that referenced
this pull request
Apr 10, 2026
Wire-sysio PR #288 (table_id namespace isolation) replaced the legacy
key_format discriminator with a per-table table_id (uint16, DJB2 hash
of the table name % 65536) computed at compile time by CDT and emitted
on every contract_row_kv_v0 SHiP delta. The new payload shape is
{ code, payer, table_id, key, value } — no more scope, table, or
key_format fields.
Two breaks need fixing on the Hyperion side:
1. ABI binary format mismatch (definitions/abi_def.ts)
table_def gained table_id (uint16) and secondary_indexes
(vector<index_def>) in wire-sysio commit a9f4614f6, and `name` was
widened from sysio::name to a free-form string. Hyperion's
AbiDefinitions used by Serialize.getTypesFromAbi() to deserialize
contract ABIs from the SHiP `account` delta still had the old
layout, so any new wire-sysio ABI would either fail to parse or
misalign every field after the table name. Update table_def to
match the chain layout exactly and add an index_def struct.
2. KV delta handler logic (workers/deserializer.ts)
The contract_row_kv handler still gated on `payload.key_format !== 0`
(a field that no longer exists) and resolved the table by scanning
the ABI for entries with non-empty key_names/key_types — fragile,
and ambiguous now that every kv::table contract has multiple such
entries. Replaced with a direct table_id lookup:
abiObj.tables.find(t => t.table_id === payload.table_id).
For scoped tables (kv_multi_index, kv::scoped_table) the ABI
key_names start with "scope" — the handler now mirrors the decoded
scope value into payload.scope and the next field into
payload.primary_key, matching the legacy contract_row layout that
downstream handlers (notably *:accounts → storeAccount → ES
table-accounts index) read from. This keeps sysio.token end-to-end
identical to before:
1. SHiP emits contract_row_kv_v0 with table_id = hash("accounts")
2. Handler resolves kvTable.name = "accounts", decodes scope=alice
and sym_code=4607832 from the BE key, decodes the asset value
via abieos.hexToJson('sysio.token', 'account', value)
3. processTableDelta routes to *:accounts
4. *:accounts extracts @accounts.amount/symbol from data.balance
5. storeAccount indexes {code, scope, amount, symbol} to ES
6. /v2/state/get_tokens reads the same fields as before
Companion wire-sysio PRs:
- Wire-Network/wire-sysio#288 (table_id namespace isolation, SHiP
contract_row_kv_v0 / contract_index_kv_v0 emit table_id)
- Wire-Network/wire-sysio#290 (unified get_table_rows API — no
Hyperion HTTP API consumer is affected: get_currency_balance is a
dedicated endpoint and the only get_table_rows caller targets a
defunct `voters` table that no longer exists on Wire chains)
- Wire-Network/wire-sysio#291 (sysio.token migrated to
kv::scoped_table — primary keys are byte-identical to
kv_multi_index so on-chain data is unchanged)
Out of scope for this commit:
- master.ts:2079 getRows()/fillCurrentStateTables() — already dead
on Wire (no sysio.system voters table). Replacement / removal is a
follow-on cleanup.
- contract_index_kv handler — remains a no-op (secondary index data
is derivative).
- addons/wirejs-native cosmetic interface updates.
heifner
added a commit
to Wire-Network/wire-libraries-ts
that referenced
this pull request
Apr 10, 2026
Wire-sysio PR Wire-Network/wire-sysio#288 widened table_def.name from sysio::name (uint64) to a free-form std::string and appended two new fields — table_id (uint16, DJB2 hash of the table name % 65536) and secondary_indexes (vector<index_def>) — for KV-table per-table namespace isolation. The wire format break means any sdk-core caller that loads a contract ABI from on-chain via get_raw_abi (or any other binary-encoded source) parses misaligned starting at the first table field — name reads the wrong bytes, every subsequent field is shifted, and ricardian_clauses ends up reading garbage. PR Wire-Network/wire-sysio#290 separately changed the get_table_rows HTTP response shape for KV-backed tables. Each row used to be the decoded struct directly (or {data, payer} when show_payer was set); the unified endpoint now returns {key, value, payer?} per row. The ChainAPI wrapper used to destructure {data, payer} on show_payer and then map rows through Serializer.decode for typed callers — both paths break against the new shape. This commit takes a hard break on the binary format (no fallback to the legacy EOSIO 8-byte name encoding) and a backward-compatible unwrap on the HTTP response shape. packages/sdk-core/src/chain/Abi.ts: - new ABI.Index interface (name, key_type, table_id) mirroring sysio::chain::index_def in wire-sysio's libraries/chain/include/sysio/chain/abi_def.hpp. - ABI.Table gains optional table_id (uint16) and secondary_indexes (Index[]) fields. Optional so hand-built test fixtures and ABIs constructed programmatically don't need to specify them; the encoder defaults to 0 / [] when omitted. - fromABI() reads name as a length-prefixed string instead of an 8-byte sysio::name uint64, then consumes table_id (uint16 LE) and secondary_indexes (vector<index_def>) after the existing type field. Also reads a trailing protobuf_types extension after enums so that any subsequent extension appended to abi_def parses cleanly to EOF. - toABI() mirrors the parser: writes table_def.name via encoder.writeString(String(table.name)) (the String() coercion keeps NameType-typed callers compiling), then table_id and secondary_indexes, and finally the empty protobuf_types string. packages/sdk-core/src/api/v1/Chain.ts: - get_table_rows() now detects the unified KV row shape — each row has both `key` and `value` keys — and unwraps to the inner value before downstream processing. The old EOSIO shapes (decoded struct directly, or {data, payer} on show_payer) are left untouched, so sdk-core remains usable against EOSIO chains. - When show_payer is set on a KV row, payer is captured into ram_payers in the same way as the legacy {data, payer} path. Missing payer fields coerce to an empty Name rather than throwing. Tests (15 new cases, 304/304 sdk-core suite green): packages/sdk-core/tests/chain/Abi.test.ts (8 cases) - Round-trips a table with table_id + empty secondary_indexes - Round-trips a long table name (>12 chars) — the whole reason name was widened - Round-trips secondary_indexes with checksum256 key_type (the case the wire-cdt #49 abigen fix unblocked end-to-end) - table_id 0 and 65535 boundary values - Missing table_id defaults to 0 on encode + decode - protobuf_types extension is consumed without affecting enums - Encoder always emits the empty protobuf_types trailer - Multi-table ABI end-to-end round-trip (structs + actions + secondary_indexes) packages/sdk-core/tests/api/v1/Chain.test.ts (7 cases) - Unwraps the new {key, value} shape into plain rows - Unwraps the new shape with show_payer + captures ram_payers - Preserves legacy plain-row shape from EOSIO chains - Preserves legacy {data, payer} show_payer shape from EOSIO - Empty rows array works for both shapes - Missing payer in new shape coerces to empty name (no throw) - Uses an in-memory MockProvider so no network access required Out of scope: - packages/sdk-core/src/resources/{Ram,Rex,Powerup}.ts call get_table_rows for tables (rammarket, rex_pool, powup_state) that do not exist in wire-sysio and have not for some time. These resources were already dead on Wire chains regardless of any of the in-flight wire-sysio PRs and remain dead after this commit. - packages/sdk-core/src/types/SystemContractTypes.ts is auto- generated and would benefit from a regen pass after sysio.token's migration to kv::scoped_table merges (Wire-Network/wire-sysio#291), but the field schemas of the structs it generates do not depend on the binary table_def layout. Companion PRs: - Wire-Network/wire-sysio#288 — chain-side table_id namespace isolation - Wire-Network/wire-sysio#290 — unified get_table_rows API - Wire-Network/wire-sysio#291 — sysio.token migrated to kv::scoped_table - Wire-Network/wire-cdt#49 — kv::scoped_table + abigen secondary_indexes - Wire-Network/wirejs-native#4 — TS schema mirror for abi_def - Wire-Network/abieos#12 — abieos C API + binary format update - Wire-Network/node-abieos#8 — bumps abieos + drops the lossy uint64 round-trip - Wire-Network/hyperion-history-api#9 — KV delta handler + AbiDefinitions
…e/kv-scoped-table
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces both old
get_table_rowsandget_kv_rowswith a single unified endpoint backed by the KV implementation. Fixes scope type ambiguity (AntelopeIO/spring#1379) andget_table_by_scopepagination loops (AntelopeIO/spring#615).Companion PR: Wire-Network/wire-cdt#49 (kv::scoped_table + upsert lambda + abigen
secondary_indexesformulti_index/kv_multi_index, plus the canonical-RecordTypechecksum256translation fix)API changes
/v1/chain/get_kv_rowsendpoint (merged intoget_table_rows)scopefield — parsed via ABIkey_types[0]with uint64 fallback (name/uint64/symbol all work, including scope=0)findfield — exact key lookup, errors if combined with bounds or if used on scoped table without scopeindex_name— accepts named indexes ("byowner") or numeric positions ("2"){key, value, payer?}objectsjson=true(was false),limit=50(was 10)key_type,encode_type,index_position) silently ignorednext_keystrips scope prefix for clean pagination round-tripsScope parsing (fixes AntelopeIO/spring#1379)
Scope is parsed using the ABI-declared type to avoid ambiguity:
"name"→ parse as name first, fall back to uint64 (handles"0"forname{})"uint64"→ parse as uint64 directly"111222211111"name-vs-uint64 ambiguityBug fix: secondary index lookups on scoped tables
The secondary index path's
fetch_primaryhelper looked up the primary row using only the secondary's storedpri_keyreference (the in-scope portion). For scoped tables (multi_index/kv_multi_index/kv::scoped_table) the actual primary key in the KV store is[scope:8B BE][pri_key], so the find missed and the row'svaluefield came back blank — and decoded keys came back as 8-byte hex blobs instead of decoded JSON. Fix: build the full primary key by prependingscope_prefix_bytesand use it for both thepri_idxfind and the row'sr.keyfield. For unscopedkv::table,scope_prefix_bytesis empty and behavior is unchanged.Clio changes
get table <code> <table>— 2 positional args (was 3 with scope)--scope,--find,--indexflagsfindwith correct ABI field names and["value"]row accessTest coverage (24 C++ test cases — added 6 since the previous push)
findexact lookup,find+bounds error,find+scoped-without-scope errorindex_namenamed and numeric, invalid index errornext_keyshow_payer, reverse, bounds, hex modeget_table_by_scopepagination no-loop (#615), reverse, reverse-paginationNew since last push (
get_table_next_key_testextended + 2 new cases):multi_indexsecondary index by name (uint64), numeric position, uint128, checksum256 (the canonical-RecordType abigen path)multi_indexsecondary pagination (limit +next_key)multi_indexsecondary reverse, hex bounds (usingkv_scoped_key_size/kv_scope_prefix_sizefor the[scope][sec]layout),show_payermulti_indexinvalidindex_name→ throwsget_table_find_scoped_test— find on scoped table without scope errors, with scope returns exact match, missing key returns 0 rowsget_table_missing_contract_test—get_table_rowson missing contract / non-existent table throws;get_table_by_scopeon missing contract returns 0 rows silentlyThe new
multi_indexsecondary tests require theget_table_testcontract's ABI to declaresecondary_indexes, which is enabled by the wire-cdt#49 abigen fix. The recompiledunittests/test-contracts/get_table_test/get_table_test.{wasm,abi}is included in this PR.Documentation
CLAUDE.mdgains a "Test Binaries" section:unit_testdoes NOT contain everything —tests/get_table_tests.cppandtests/test_*.cppbuild intoplugin_test. Standard pre-PR sweep isunit_test,plugin_test,contracts_unit_test, andtest_fc.Also updates Python integration tests (
queries.py,get_kv_rows_test.py,nodeop_run_test.py,nodeop_lib_test.py,nested_container_multi_index_test.py,plugin_http_api_test.py) and docs (get-table-rows-api.md).