Feature #7024

[Keep] keepstore status.json should report runtime environment

Added by Tom Clegg over 3 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Keep
Target version:
Start date:
08/19/2015
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

Some useful/easy things to report in /status.json that can be useful for troubleshooting:
  • GOGC env (or 100 if not set) - see https://golang.org/pkg/runtime/debug/#SetGCPercent
  • GOMAXPROCS env
  • Startup and current timestamps (in nanoseconds, like the times in MemStats)
  • A list of all the operations in progress, with these fields: operation (GET/PUT/HEAD/...), hash, remote ip, how long since it was received (or timestamp received), requests state (reading request body, checking for existing copy on disk, writing to disk, etc)
  • For each integer x in 0..600, number of requests received and number of requests completed as of x seconds ago, bucketed by handler (head, get, put, index, status).

History

#1 Updated by Brett Smith over 3 years ago

  • Story points deleted (0.5)

Clearing the story points with the expectation that there are going to be many requests added in the next couple of days. We can triage those requests and re-point later.

#2 Updated by Ward Vandewege over 3 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg over 3 years ago

Some debug information will require us to start hiding (at least some of) the status report behind an admin token, or setting up a separate http server on a different port that's firewalled off appropriately. For example, "URLs of all current requests" would be useful but can't be shown to non-admin clients. The "different port" option lets us access a bunch of debugging tools from the Go community (e.g., pprof) that are trivial to install: they just set themselves up at conventional paths using the default http server in net/http.

#4 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#5 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#6 Updated by Tom Clegg over 3 years ago

  • Description updated (diff)

#7 Updated by Brett Smith over 2 years ago

Additional things to report:

It should be possible to tell what generated a status.json even if you've forgotten about the original request:

  • Service name (keepstore, crunch-dispatch, etc.)
  • Hostname
  • Timestamp

Basic status:

  • Daemon version
  • Operational status

Also available in: Atom PDF