Logarithmic scale when plotting results

[Magalame]

Hi!
I was wondering if there was such a feature in criterion, and if not how would I go about implementing it in the library?

[RyanGlScott]

criterion currently doesn't support configuring logarithmic scales to my knowledge, no.

As for how difficult it would be to implement such a feature, I don't know off the top of my hand. That being said, the plotting functionality in this criterion makes use of the flot JavaScript library, so if flot supports logarithmic scales, there may be a way to incorporate that functionality into criterion as well.

[Lythenas]

According to flot documentation you can use something like this:

xaxis: {
    transform: function (v) { return Math.log(v); },
    inverseTransform: function (v) { return Math.exp(v); }
}

in the javascript (in the html template).

Note that I haven't tested this.

I also think while the criterion report looks pretty it should have more features / be more easily extendable. E.g able to split the overview graph at the top into groups. But maybe that is better implemented in another package and not in criterion directly.

[RyanGlScott]

Thanks for the tip, @Lythenas. A next step will be figuring out what kind of interface we should offer for tweaking the axes' scales in the HTML report itself. One option is to accept extra command-line arguments for configuring this. Alternatively, we could try offering this UI in the report itself, although that would be much more work to implement.

[Lythenas]

I not sure what the best approach is. I think a general interface to write custom report exporters would be the easiest way to expand criterion. But I would keep the default exporter as is (by default) because for basic benchmarks it is fine IMO.

I wouldn't make this configurable by command line because it will likely get very complicated very fast. Maybe a list of presets would be ok as command line arguments.

I'm thinking criterion could offer something like this:

class ReportExporter where
  formatReport :: [Report] -> Text

(and maybe helpers to extend the default flot template)

And I could write my own exporter that generates HTML or custom JSON or whatever I want.

FYI: hvega looks like it has a nice DSL to generate data for Vega-Lite (which is an alternative to flot). Maybe we could build something similar for flot or replace flot with vega lite (maybe in an alternative exporter).

[jberryman]

There should be a simple heuristic to choose between a linear scale or a log scale when the bar plot would otherwise be useless, which is most of the time in my experience

[cartazio]

i think log scale should probably be the default ... maybe wiht the base "ofset" at the low end at pico sends? or we shift to box and whisker plots? on log scale https://en.wikipedia.org/wiki/Box_plot

maybe @Hasufel or @Bodigrim have thoughts? (theres also the issue that default rendering js last i looked doesn't just consume summary statistics, but rather all the points, last i looked? or some other insane algorithmic issue )

[Bodigrim]

There should be a simple heuristic to choose between a linear scale or a log scale when the bar plot would otherwise be useless, which is most of the time in my experience

What kind of heuristic could it be? I cannot think of anything simple and one-size-fits-all.

i think log scale should probably be the default

Use cases for benchmarking are quite diverse. Like, if we benchmark different implementations of an algorithm, then log-scale usually looks very sensible, because it nicely captures differences in asymptotical growth without making plot unreadable. But if I benchmark the very same function on different inputs, I would definitely prefer a normal scale.

or we shift to box and whisker plots?

Absolutely, it would be nice to have a box-and-whisker plot. But altogether it looks to me that this is not a job for criterion. There are many ways to transform a statistical data, far more than it is sensible to configure from criterion's command line.

I remember that criterion can dump results as JSON. Isn't it enough? External tools are free to analyse and visualise it in any imaginable way.

[Magalame]

I think the linear scale should be the default, I think it's kind of a general assumption regarding any plot

[cartazio]

I respectfully 100% disagree and I am more than happy to have an educational resource collectiong/back and forth email/whatever with both of you to make this case.

I realize that people dont see log scale plots till relatively late in their educations, but for the way people use criterion: namely to compare different algorithms, we care about the multiplicative difference between two algorithms on the same input (2x or 10x or whatever speed difference). Default linear scales makes the emphasis on whatever algorithm is slowest. and complicate actually visually seeing multiplicative difference in perf.

(who using criterion hasn't commented out the slowest algorithm on their machine so the visual scale "focuses" on the actual contenders?). I'm more than happy to also provide clear opinion references from statistics/applied statistics/public health where correct interpretable data plotting is part of the core decision making loop.

I'm wiling to be convinced i'm wrong, but I ask that you give me the opportunity to convince you and address whatever grounded reasons you have in mind.

[cartazio]

(also i'm interested in making this happen :) )

[Bodigrim]

but for the way people use criterion: namely to compare different algorithms, we care about the multiplicative difference between two algorithms on the same input (2x or 10x or whatever speed difference).

If we use criterion to compare different algorithms, then yes. My argument is that this is not always the case. For instance, I often test the same function on different sizes of input (like, [100,110..200]), for which I would very much prefer a normal scale.

who using criterion hasn't commented out the slowest algorithm on their machine so the visual scale "focuses" on the actual contenders?

:D Nice!

I totally agree that having an option to switch to log-scale would be awesome. I would not make it a default however.

Maybe we can have embed a switch directly into HTML page? So that users can change between normal and log scales without rerunning benchmarks? It should not be that difficult; I am happy to revive my webdev skills if we choose this avenue.

[cartazio]

thats an even better idea! (on page switch/toggle). In that case for log scale probably should have it start a "picoseconds" so there some margin things start with ? idk

(also last i checked years ago) : I do think that the default html template has(?) some algorithm issues (either DOM related or because the raw data set is dumped into the page instead of just order statistics + mean + variance + min + max).

@Bodigrim you, as usual, make great points!

[RyanGlScott]

The ability to toggle between linear and log scales via a switch on the HTML page itself sounds like a promising way forward. This avoids one of the problems that a command-line–only switch would have, since it would allow toggling the scales on a per-chart basis.