Arvados

From what I'm reading on lograge's@ github page (https://github.com/roidrage/lograge - FAQ section), we can include exceptions on the custom_options configuration payload. We just have to make sure to convert the stack trace to a single line.

Another mention on this topic: https://github.com/roidrage/lograge/issues/190

Updates at 25d42fc - branch 15319-api-useful-stacktraces
Test run: https://ci.curoverse.com/job/developer-run-tests/1394/ (testing the commit previous to merging a broken master)

At first and from the grooming investigation, this story seemed pretty trivial as the lograge documentation states that including the exeptions on the log is a supported feature, but in our particular case it doesn't work because we catch all exceptions with a rescue_from(Exception, ..., :with => :render_error) on application_controller.rb, and the exception handlers cannot propagate the exceptions being handled by re-raising them, so lograge doesn't receive them.

The solution found was assigning the exception information to Thread.current and consuming it at lograge's custom formatter.

The other tricky part was testing by using stubs and mocha expectations.

Lucas Di Pentima wrote:

Updates at 25d42fc - branch 15319-api-useful-stacktraces
Test run: https://ci.curoverse.com/job/developer-run-tests/1394/ (testing the commit previous to merging a broken master)

At first and from the grooming investigation, this story seemed pretty trivial as the lograge documentation states that including the exeptions on the log is a supported feature, but in our particular case it doesn't work because we catch all exceptions with a rescue_from(Exception, ..., :with => :render_error) on application_controller.rb, and the exception handlers cannot propagate the exceptions being handled by re-raising them, so lograge doesn't receive them.

The solution found was assigning the exception information to Thread.current and consuming it at lograge's custom formatter.

The other tricky part was testing by using stubs and mocha expectations.

The test build had workbench errors so I submitted another build: https://ci.curoverse.com/view/Developer/job/developer-run-tests/1396/

The solution looks good. Could we add a documentation page in the admin section about 'how to diagnose errors' that mentions these commands.

To get a specific request:

grep req-frdyrcgdh4rau1ajiq5q production.log | jq .

To get the backtrace from a failed request:

grep req-frdyrcgdh4rau1ajiq5q production.log | jq -r .exception_backtrace

Please take into account that the test run 1396 is using a commit that includes current master, that's why the api server tests failed.

Updates at 783b9cc83

Adds documentation describing #15318 and #15319 features with an example.

Lucas Di Pentima wrote:

Updates at 783b9cc83

Adds documentation describing #15318 and #15319 features with an example.

This is great!

My only other suggestion is to add links to the troubleshooting page from the install documentation, because it might be hard to find where it is now. Otherwise LGTM, please merge.