Troubleshooting aids¶

Troubleshoot usage problems:

• Improve error messages (e.g., clients should not crash and dump stack when a server is slow/unresponsive)

Troubleshoot compute nodes/images:

• Idea #21581: Crunch saves compute node journals to collections readable only by administrators
• Idea #21424: Way to run a diagnostic container that captures all system logs, not just Crunch's

Troubleshoot arvados system services:

• Save snapshot of internals (goroutines / memory profile) of specified system service(s) to a collection, and provide instructions for viewing
• Save last N minutes of logs from all arvados services running on this host
• Turn on debug mode temporarily, without restarting services
• Option to send railsapi logs to journal or stderr
• Option to send go service logs to journal directly (structured, including loglevel)

Expose config/scaling issues:

• Scan metrics for recent "near/at capacity" signals
• Probe for proper nginx/proxy config (e.g., max request body size)