Feature #7024
open
[Keep] keepstore status.json should report runtime environment
Added by Tom Clegg over 9 years ago.
Updated 9 months ago.
Release relationship:
Auto
Description
Some useful/easy things to report in
/status.json that can be useful for troubleshooting:
- GOGC env (or 100 if not set) - see https://golang.org/pkg/runtime/debug/#SetGCPercent
- GOMAXPROCS env
- Startup and current timestamps (in nanoseconds, like the times in MemStats)
- A list of all the operations in progress, with these fields: operation (GET/PUT/HEAD/...), hash, remote ip, how long since it was received (or timestamp received), requests state (reading request body, checking for existing copy on disk, writing to disk, etc)
- For each integer x in 0..600, number of requests received and number of requests completed as of x seconds ago, bucketed by handler (head, get, put, index, status).
- Story points deleted (
0.5)
Clearing the story points with the expectation that there are going to be many requests added in the next couple of days. We can triage those requests and re-point later.
- Description updated (diff)
Some debug information will require us to start hiding (at least some of) the status report behind an admin token, or setting up a separate http server on a different port that's firewalled off appropriately. For example, "URLs of all current requests" would be useful but can't be shown to non-admin clients. The "different port" option lets us access a bunch of debugging tools from the Go community (e.g., pprof) that are trivial to install: they just set themselves up at conventional paths using the default http server in net/http.
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
Additional things to report:
It should be possible to tell what generated a status.json even if you've forgotten about the original request:
- Service name (keepstore, crunch-dispatch, etc.)
- Hostname
- Timestamp
Basic status:
- Daemon version
- Operational status
- Target version deleted (
Arvados Future Sprints)
- Target version set to Future
Also available in: Atom
PDF
Updated by 
Updated by 
Updated by 
Updated by