feat(storer): use buf chan to improve perfomance #5170
Conversation
This reverts commit a57b211.
|
The metrics show a clear improvement when comparing the first and last batch of data. Using buffered channels reduces processing time by up to 30%. Additionally, sampling time is mostly concentrated at the lower bound, suggesting that the time reduction could exceed 30%. 
Testing was done on |
|
|
||
| chunkC := make(chan *reserve.ChunkBinItem) | ||
| workers := max(4, runtime.NumCPU()) | ||
| chunkC := make(chan *reserve.ChunkBinItem, 3*workers) |
There was a problem hiding this comment.
the value 3 and then 3*workers were chosen after some testing and benchmarking? How did you choose to use these?
There was a problem hiding this comment.
The main reason is to prevent workers from waiting for tasks. It decouples the producer (finding chunks) from the consumers (the workers), allowing it to queue up just enough work to ensure the workers are always busy. I will try to repeat the tests with greater values to double check. But this seems good enough for current logic.
Checklist
Description
Improves making sample time for redistribution game from 15-30% on bee-light-testnet.
Open API Spec Version Changes (if applicable)
Motivation and Context (Optional)
Related Issue (Optional)
Screenshots (if appropriate):