Add support for metrics with Prometheus

[hairyhenderson]

(edited for brevity - see previous edits if you're interested in the history!)

Fixes #3390

At long last I've found time to work on this. Sorry about the delay - it's been a bit of a rough summer!

This adds PR does a number of things:

• adds a new admin.api.metrics module for scraping metrics on the admin API endpoint at /metrics. This is always mounted as long as the admin API is enabled, giving Caddy automatic support for Prometheus metrics.
• adds a new http.handlers.metrics module for configuring which port/path metrics should be scraped on. This will be mounted as-needed, and can be used when the admin API is disabled. This module may get more configurable in future PRs.
• instruments all HTTP Middleware modules to track request/error counts, request duration, time-to-first-byte, request size, and response size
• instruments all Admin handlers to track request and error counts
• instruments the Go runtime with the Go Collector
• exports a go_build_info metric (see prometheus.NewBuildInfoCollector)
• adds a new metrics directive to Caddyfile

It's worth noting also that this fully enables any module author to register and observe their own custom Prometheus metrics. I can add a brief tutorial in the docs at some point to that end (in short: just register them with prometheus.DefaultRegisterer).

Many thanks to @mholt who helped me understand heaps more about the Caddy internals and make decisions on how to put the modules together!

Expand to see a sample (warning: verbose!)

Config file:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":9180"],
          "routes": [
            { 
              "match": [{"path":["/metrics"]}],
              "handle": [{ "handler": "metrics" }]
            },
            {
              "match": [{ "path": ["/foo", "/bar"] }],
              "handle": [{"handler": "static_response", "body": "Hello world!\n"}]
            },
            {
              "match": [{ "path": ["/error"] }],
              "handle": [{ "handler": "error" }]
            },
            {
              "match": [{ "path": ["*"]}],
              "handle": [{"handler":"static_response", "status_code": "404"}]
            }
          ]
        }
      }
    }
  }
}

Doing a few requests to seed some metrics observations:

$ curl localhost:9180/foo                                                         
Hello world!
$ curl localhost:9180/error
$ curl localhost:9180/baz

Gather the metrics from the admin API, specifying OpenMetrics format (optional):

$ curl -H "Accept: application/openmetrics-text;version=0.0.1" localhost:2019/metrics
# HELP caddy_http_request_duration_quantile_seconds Summary of round-trip request durations.
# TYPE caddy_http_request_duration_quantile_seconds summary
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.1"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.5"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.95"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.99"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.999"} 8.1123e-05
caddy_http_request_duration_quantile_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 8.1123e-05
caddy_http_request_duration_quantile_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.1"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.5"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.95"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.99"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.999"} 0.000112656
caddy_http_request_duration_quantile_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 0.000112656
caddy_http_request_duration_quantile_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_request_duration_seconds Histogram of round-trip request durations.
# TYPE caddy_http_request_duration_seconds histogram
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.01"} 1 # {method="GET",server="srv0",handler="static_response",code="200"} 8.1123e-05 1.599930472138724e+09
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="3.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="8.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="20.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="60.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="120.0"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_duration_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 8.1123e-05
caddy_http_request_duration_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.01"} 1 # {handler="static_response",code="404",method="GET",server="srv0"} 0.000112656 1.59993048773067e+09
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="3.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="8.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="20.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="60.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="120.0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_duration_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 0.000112656
caddy_http_request_duration_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_request_errors Number of requests resulting in middleware errors.
# TYPE caddy_http_request_errors counter
caddy_http_request_errors_total{handler="error",server="srv0"} 1.0
# HELP caddy_http_request_size_bytes Total size of the request. Includes body
# TYPE caddy_http_request_size_bytes histogram
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="256.0"} 1 # {method="GET",server="srv0",handler="static_response",code="200"} 59.0 1.599930472138773e+09
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1024.0"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4096.0"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="16384.0"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="65536.0"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="262144.0"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_size_bytes_sum{code="200",handler="static_response",method="GET",server="srv0"} 59.0
caddy_http_request_size_bytes_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="256.0"} 1 # {server="srv0",handler="static_response",code="404",method="GET"} 59.0 1.59993048773072e+09
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1024.0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4096.0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="16384.0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="65536.0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="262144.0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_size_bytes_sum{code="404",handler="static_response",method="GET",server="srv0"} 59.0
caddy_http_request_size_bytes_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_requests_in_flight Number of requests currently handled by this server.
# TYPE caddy_http_requests_in_flight gauge
caddy_http_requests_in_flight{handler="error",server="srv0"} 0.0
caddy_http_requests_in_flight{handler="static_response",server="srv0"} 0.0
# HELP caddy_http_requests Counter of HTTP(S) requests made.
# TYPE caddy_http_requests counter
caddy_http_requests_total{handler="error",server="srv0"} 1.0
caddy_http_requests_total{handler="static_response",server="srv0"} 2.0
# HELP caddy_http_response_latency_quantile_seconds Summary of times to first byte in response bodies.
# TYPE caddy_http_response_latency_quantile_seconds summary
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.1"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.5"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.95"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.99"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.999"} 5.163e-06
caddy_http_response_latency_quantile_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 5.163e-06
caddy_http_response_latency_quantile_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.1"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.5"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.95"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.99"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.999"} 3.541e-06
caddy_http_response_latency_quantile_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 3.541e-06
caddy_http_response_latency_quantile_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_response_latency_seconds Histogram of times to first byte in response bodies.
# TYPE caddy_http_response_latency_seconds histogram
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.01"} 1 # {method="GET",server="srv0",handler="static_response",code="200"} 5.163e-06 1.599930472138651e+09
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="3.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="8.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="20.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="60.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="120.0"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_latency_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 5.163e-06
caddy_http_response_latency_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.01"} 1 # {handler="static_response",code="404",method="GET",server="srv0"} 3.541e-06 1.599930487730565e+09
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="3.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="8.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="20.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="60.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="120.0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_latency_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 3.541e-06
caddy_http_response_latency_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_response_size_bytes Size of the returned response.
# TYPE caddy_http_response_size_bytes histogram
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="256.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1024.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4096.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="16384.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="65536.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="262144.0"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_size_bytes_sum{code="200",handler="static_response",method="GET",server="srv0"} 13.0
caddy_http_response_size_bytes_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="256.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1024.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4096.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="16384.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="65536.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="262144.0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_size_bytes_sum{code="404",handler="static_response",method="GET",server="srv0"} 0.0
caddy_http_response_size_bytes_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP go_build_info Build information about the main Go module.
# TYPE go_build_info gauge
go_build_info{checksum="",path="github.com/caddyserver/caddy/v2",version="(devel)"} 1.0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0.0"} 3.2547e-05
go_gc_duration_seconds{quantile="0.25"} 3.2547e-05
go_gc_duration_seconds{quantile="0.5"} 6.3396e-05
go_gc_duration_seconds{quantile="0.75"} 7.4878e-05
go_gc_duration_seconds{quantile="1.0"} 7.4878e-05
go_gc_duration_seconds_sum 0.000170821
go_gc_duration_seconds_count 3
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 15.0
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15"} 1.0
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 5.022168e+06
# HELP go_memstats_alloc_bytes Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes counter
go_memstats_alloc_bytes_total 8.611928e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448027e+06
# HELP go_memstats_frees Total number of frees.
# TYPE go_memstats_frees counter
go_memstats_frees_total 23676.0
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.03498476581089659
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.480864e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 5.022168e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.9301888e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.692864e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 37797.0
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.8294272e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5994752e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.599930464748596e+09
# HELP go_memstats_lookups Total number of pointer lookups.
# TYPE go_memstats_lookups counter
go_memstats_lookups_total 0.0
# HELP go_memstats_mallocs Total number of mallocs.
# TYPE go_memstats_mallocs counter
go_memstats_mallocs_total 61473.0
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888.0
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384.0
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 124848.0
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072.0
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.51016e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.657349e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.114112e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.114112e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.584256e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 12.0
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1.0
# HELP promhttp_metric_handler_requests Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests counter
promhttp_metric_handler_requests_total{code="200"} 0.0
promhttp_metric_handler_requests_total{code="500"} 0.0
promhttp_metric_handler_requests_total{code="503"} 0.0
# EOF

Gathering metrics from the metrics handler. Note that admin_http_requests_total is now present (since the last call to the admin metrics endpoint):

$ curl localhost:9180/metrics                                                     
# HELP admin_http_requests_total Counter of requests made to admin endpoints.
# TYPE admin_http_requests_total counter
admin_http_requests_total{code="200",handler="metrics",method="get",path="/metrics"} 1
# HELP caddy_http_request_duration_quantile_seconds Summary of round-trip request durations.
# TYPE caddy_http_request_duration_quantile_seconds summary
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.1"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.5"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.95"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.99"} 8.1123e-05
caddy_http_request_duration_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.999"} 8.1123e-05
caddy_http_request_duration_quantile_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 8.1123e-05
caddy_http_request_duration_quantile_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.1"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.5"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.95"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.99"} 0.000112656
caddy_http_request_duration_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.999"} 0.000112656
caddy_http_request_duration_quantile_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 0.000112656
caddy_http_request_duration_quantile_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_request_duration_seconds Histogram of round-trip request durations.
# TYPE caddy_http_request_duration_seconds histogram
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.01"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="3"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="8"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="20"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="60"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="120"} 1
caddy_http_request_duration_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_duration_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 8.1123e-05
caddy_http_request_duration_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.01"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="3"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="8"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="20"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="60"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="120"} 1
caddy_http_request_duration_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_duration_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 0.000112656
caddy_http_request_duration_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_request_errors_total Number of requests resulting in middleware errors.
# TYPE caddy_http_request_errors_total counter
caddy_http_request_errors_total{handler="error",server="srv0"} 1
# HELP caddy_http_request_size_bytes Total size of the request. Includes body
# TYPE caddy_http_request_size_bytes histogram
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="256"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1024"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4096"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="16384"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="65536"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="262144"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_request_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_size_bytes_sum{code="200",handler="static_response",method="GET",server="srv0"} 59
caddy_http_request_size_bytes_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="256"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1024"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4096"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="16384"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="65536"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="262144"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_request_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_request_size_bytes_sum{code="404",handler="static_response",method="GET",server="srv0"} 59
caddy_http_request_size_bytes_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_requests_in_flight Number of requests currently handled by this server.
# TYPE caddy_http_requests_in_flight gauge
caddy_http_requests_in_flight{handler="error",server="srv0"} 0
caddy_http_requests_in_flight{handler="metrics",server="srv0"} 1
caddy_http_requests_in_flight{handler="static_response",server="srv0"} 0
# HELP caddy_http_requests_total Counter of HTTP(S) requests made.
# TYPE caddy_http_requests_total counter
caddy_http_requests_total{handler="error",server="srv0"} 1
caddy_http_requests_total{handler="static_response",server="srv0"} 2
# HELP caddy_http_response_latency_quantile_seconds Summary of times to first byte in response bodies.
# TYPE caddy_http_response_latency_quantile_seconds summary
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.1"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.5"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.95"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.99"} 5.163e-06
caddy_http_response_latency_quantile_seconds{code="200",handler="static_response",method="GET",server="srv0",quantile="0.999"} 5.163e-06
caddy_http_response_latency_quantile_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 5.163e-06
caddy_http_response_latency_quantile_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.1"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.5"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.95"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.99"} 3.541e-06
caddy_http_response_latency_quantile_seconds{code="404",handler="static_response",method="GET",server="srv0",quantile="0.999"} 3.541e-06
caddy_http_response_latency_quantile_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 3.541e-06
caddy_http_response_latency_quantile_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_response_latency_seconds Histogram of times to first byte in response bodies.
# TYPE caddy_http_response_latency_seconds histogram
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.01"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="3"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="8"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="20"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="60"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="120"} 1
caddy_http_response_latency_seconds_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_latency_seconds_sum{code="200",handler="static_response",method="GET",server="srv0"} 5.163e-06
caddy_http_response_latency_seconds_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.01"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.05"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.1"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.2"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="0.4"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="3"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="8"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="20"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="60"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="120"} 1
caddy_http_response_latency_seconds_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_latency_seconds_sum{code="404",handler="static_response",method="GET",server="srv0"} 3.541e-06
caddy_http_response_latency_seconds_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP caddy_http_response_size_bytes Size of the returned response.
# TYPE caddy_http_response_size_bytes histogram
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="256"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1024"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4096"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="16384"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="65536"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="262144"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_response_size_bytes_bucket{code="200",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_size_bytes_sum{code="200",handler="static_response",method="GET",server="srv0"} 13
caddy_http_response_size_bytes_count{code="200",handler="static_response",method="GET",server="srv0"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="256"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1024"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4096"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="16384"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="65536"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="262144"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="1.048576e+06"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="4.194304e+06"} 1
caddy_http_response_size_bytes_bucket{code="404",handler="static_response",method="GET",server="srv0",le="+Inf"} 1
caddy_http_response_size_bytes_sum{code="404",handler="static_response",method="GET",server="srv0"} 0
caddy_http_response_size_bytes_count{code="404",handler="static_response",method="GET",server="srv0"} 1
# HELP go_build_info Build information about the main Go module.
# TYPE go_build_info gauge
go_build_info{checksum="",path="github.com/caddyserver/caddy/v2",version="(devel)"} 1
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.2547e-05
go_gc_duration_seconds{quantile="0.25"} 5.7028e-05
go_gc_duration_seconds{quantile="0.5"} 6.3396e-05
go_gc_duration_seconds{quantile="0.75"} 7.4878e-05
go_gc_duration_seconds{quantile="1"} 8.0225e-05
go_gc_duration_seconds_sum 0.000308074
go_gc_duration_seconds_count 5
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 14
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.29972e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 8.740632e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.448027e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 31171
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 5.8539779885652035e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.511584e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.29972e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.9891712e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.43072e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 31302
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.8187776e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6322432e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5999307179413178e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 62473
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 124848
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.434896e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.626629e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 786432
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 786432
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.584256e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 14
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

Signed-off-by: Dave Henderson dhenderson@gmail.com

[SuperQ]

Most of the metric naming is pretty good. A couple of comments.

[SuperQ]

One thing I would like to promote is that we keep the configuration as simple as possible to start with. There's no need to have too many knobs to control metric output without having some good justification for those knobs.

The more we make metrics output consistent, the easier it is to support. Both in terms of code, but also in terms of dashboards, alerts, and comparing one Caddy install to another.

[hairyhenderson]

Thanks for looking at this, @SuperQ!

There's no need to have too many knobs to control metric output without having some good justification for those knobs.

Yup, I agree. I'll defer any to a future PR.

[hairyhenderson]

Ok, I've done a bunch of rework since the original PR, so PTAAL @mholt @SuperQ if you don't mind 🙂

Caddyfile support is now in too, so I feel like this is ready!

[mholt]

Very nice so far! I'm looking forward to getting this in, as are many others I'm sure.

Just did a high level first pass.

As I don't have experience with Prometheus, would it be possible to get at least one other experienced Prometheus user to do a review before we merge it? If you can find somebody and point them this way, that would be ideal. I can tweet about it too if you'd like.

[SuperQ]

As I don't have experience with Prometheus, would it be possible to get at least one other experienced Prometheus user to do a review before we merge it?

Prometheus person here, already looking at this. 😄 Maybe also @beorn7 or @roidelapluie would be able to comment.

[beorn7]

I'm happy to have a look. Just super busy these days. Hopefully tomorrow…

[hairyhenderson]

Thanks for the review @SuperQ @mholt @francislavoie - I've addressed most of the feedback, just a few outstanding comments where I'm waiting for feedback 🙂. PTAAL!

[mholt]

Looking good, we're almost there! I just have a few remaining suggestions and questions.

[beorn7]

Will still try to give this a review tomorrow…

[mholt]

LGTM!

I will await a final pass (or however many are necessary) or OK from the other reviewer(s) before merging this.

My last two suggestions here are just totally unimportant nits, doesn't matter to me either way.

[hairyhenderson]

Ok, I've squashed it all down to a single commit - PTAL @mholt @SuperQ - and @beorn7 please speak up if you wanted to review this today, otherwise we can merge and follow up with any changes 😉

[beorn7]

Thanks for all the great discussions. I have reached the end of the day, and again didn't find time to look at this in detail. I guess you should just merge now, and I'll earmark this for looking at this later and will let you know if I have any follow-ups to suggest.

[beorn7]

And BTW, super happy to see Prometheus metrics in Caddy. I kind of thought that would be a great thing to have ever since I noticed Caddy.

[roidelapluie]

This is good but I think we should initialize some of the counters with the most common http codes.

[hairyhenderson]

This is good but I think we should initialize some of the counters with the most common http codes.

Thanks @roidelapluie - that's a good point. I'll keep this for a follow-up PR I think. There are a number of dynamic labels and I want to be smart about how I initialize all the various combinations...

[mholt]

I'll merge this in a couple hours unless there are any objections. Thanks all, for your reviews!

[SuperQ]

LGTM

[hairyhenderson]

Thanks all for the reviews and guidance! I really appreciated it. Very excited to have this in finally 😂 🎉