-
Notifications
You must be signed in to change notification settings - Fork 72
fix: Multiple store read #552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: edge
Are you sure you want to change the base?
fix: Multiple store read #552
Conversation
|
Tests pass with the same failures reported in #553 |
fcead56 to
101961c
Compare
| case hb_store:list(Store, << ID/binary, "/commitments">>) of | ||
| {ok, []} -> [ID]; | ||
| {ok, CommitmentIDs} -> CommitmentIDs; | ||
| _ -> [] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't see a use case to return an empty array. The LMDB store didn't return not_found, but the hb_store_fs did, so swapping these two stores in the dev_query_test_vectors:simple_blocks_query_test will make the test fail (which shouldn't happen).
| Store = hb_opts:get(store, no_viable_store, Opts), | ||
| UncommittedID = hb_message:id(Msg, none, Opts#{ linkify_mode => discard }), | ||
| %% TODO: Confirm tha this should only look into local stores only | ||
| Store = hb_store:scope(hb_opts:get(store, no_viable_store, Opts), local), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
On edge, because LMDB store list was returning {ok, []} when it didn't find the information, the not_found case was never executed, making this request to not propagate to other stores.
This can impact performance, since now it will request to the configured stores. By limiting the scope to local we can mitigate this performance impact, but we need to be aware it will always be worse than just requesting to LMDB.
|
|
||
| cache_suite_test_() -> | ||
| hb_store:generate_test_suite([ | ||
| {"store ans104 message", fun test_store_ans104_message/1}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test wasn't being used (maybe deleted by mistake?). Re-add it.
| ), | ||
| Msg#{ <<"commitments">> => NewCommitments }. | ||
|
|
||
| read_all_commitments_by_store(Msg, Store, Opts) when not is_list(Store) -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With read_all_commitments_by_store we make sure all actions only occour in one Store, and not retried in different stores if the item isn't found in the first one.
| hb_store:write(Store2, <<"data/final_id2">>, <<"7890">>), | ||
| %% Link | ||
| %% TODO: Not sure if this structure is possible in HB | ||
| hb_store:make_link(Store2, <<"data/final_id2">>, <<"group1/group12/data">>), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if this structure is possible in HB, but in this scenario, the old resolve followed by list would fail.
This solves the issue where requesting
http://localhost:8734/<ID>doesn't return the rightcontent-type(and other fields).The function hb_cache:store_read executes different actions (different
hb_storecalls) for each store. This will break the logic when callingtypein one store and thenlistin another store, which always returns the ok tuple (hb_store_lmdb).not_found, then call type on the second store, which returnscomposite.liston the first store, which always returns (even if empty), not accessing the value we want in the second store.This PR changes this, so
hb_cache:store_readprocesses the stores likehb_store, to run each logic separately for each store.The change to
lmdbfile and thehb_cache:listfunction is optional, since I didn't detect an error with it, but this behaviour doesn't make sense because it is equal to thehb_cache:store_read.resolvebehaviourWe have two behaviours for
resolvefunction:binary(resolved or unresolved path) ornot_foundatom inhb_store_fs.binary(resolved or unresolved path) inhb_store_lmdbnot_found.binary(resolved or unresolved path) or{error, any()}inhb_store_rocksdb.not_foundis never returned.The different behaviour between them is the reason to have different
resolvehandling inresolved_typeandresolved_list. If we agree on modifying the return to always containnot_found(which may imply reading the object), this might not be needed.resolved_readnot implementedhb_store_fs,hb_store_lmdb, andhb_store_lrualready runresolvewhenreadis called.hb_store_fsandhb_store_lru-read/2resolves the key before trying to access it.hb_store_lmdb- tries to read first directly. If not successful, it will callresolve_path_links.We can remove
resolvecall before eachread, and trust the store to call it, or implement theresolved_readto make sure theresolveandreadare always called for the same store.Why not make
typeandlistinternally callresolve?There are a few cases where
typeandlistare used whereresolvedisn't needed or made in a different place of the function call.Cases not changed from
resolveandlisttoresolved_listhb_cache:read_all_commitmentsuses theCommitmentsPathinside its logic, not being able to collapse intoresolved_list.