Commits on Jul 9, 2025

1. One file per process: move file management from MetricStore to `Dir… …

   …ectFileStore`
   
   The overall intention here is to have one file per process, instead of one file per process per metric.
   
   We originally went with separate files per metric to make "observe"s faster for multi-threaded processes.
   Making the file lock contention be per-metric instead of per-process would theoretically improve performance
   by reducing waits on that lock if two threads wanted to observe at the same time.
   
   In practice, however, the result is that we end up with too many files, which causes problems with folks with
   large numbers of metrics, and also reading lots of small files instead of a few large ones makes exports slow.
   
   The overall mechanic to achieve this is to add one more special reserved label to all labelsets: ___metric_name
   Once the metric name is in the labelset, we can store everything in one file per process.
   
   In the code, this means mostly moving a bunch of the methods dealing with "which files to open" from the per-metric
   class (`MetricStore`) to the "global" store class `DirectFileStore`. This allows us to maintain all external interfaces
   the same, and tests passing, while "naively" achieving the objective.
   
   In practice, this has a HUGE drawback. When exporting, we're not reading all the (now bigger) files once for every metric and
   discarding most of the values we read. I expect this will make exporting outrageously slow.
   
   The answer to this will come on the followup commits, where we'll make the exporter less "elegant" and less "store agnostic",
   but hopefully much faster.

   

   dmagliola committed Jul 9, 2025
   Configuration menu

   • View commit details
   • Copy full SHA for aae3d99
   • Browse repository at this point

   Copy the full SHA

   aae3d99 View commit details

   Browse the repository at this point in the history

2. Export the DirectFileStore metrics without having to read files multi… …

   …ple times
   
   This is absolutely disgusting, but it's the kind of thing we need to do to get this to work.
   
   There are 2 disgusting parts:
   - In the text formatter, we need to read the files directly, and aggregate all the values ourselves
     without going through MetricStore / each metric's individual store. This is pretty horrendous, but
     I think we don't have a way around this.
   - Each Metric exposes "values" by reading those values from an internal store. HOWEVER, Summary and Histogram
     do some very clever, very pretty hackery to return a hash instead of a single value, for their buckets, sum, count, etc.
   
   This last thing breaks HORRENDOUSLY in this model, because we need to PASS IN the values we have aggregated, instead of
   letting them go down to their internal stores. That's terrible as an interface to expose to our users, which will probably
   not be calling this method, but if they do, it's horrible.
   
   We probably need to extract that part for the purposes of formatting, either to the Text formatter, or some intermediary object.
   If we do this, either we duplicate the clever code, or users stop having a chance to get those nice values, so we should probably duplicate.
   
   ORRRRR, we completely restructure how everything works. We'd mostly be changing internal interfaces, that most users won't see...
   But I don't have a concrete proposal to how to do that.

   

   dmagliola committed Jul 9, 2025
   Configuration menu

   • View commit details
   • Copy full SHA for b5190f8
   • Browse repository at this point

   Copy the full SHA

   b5190f8 View commit details

   Browse the repository at this point in the history