Bug #16391
closedimprove container logging when c-d-s has a problem with sbatch execution
Description
When crunch-dispatch-slurm runs into a problem with sbatch, e.g.
"sbatch: error: Batch job submission failed: Requested node configuration is not available"
that error is visible in c-d-s' logs. But in the container logs we print:
error submitting container to slurm: "PANIC=runtime error: invalid memory address or nil pointer dereference"
We need to bubble up the real error message.
This was observed on 1.4.3, presumably the error still exists on 2.0.2, but I haven't checked that.
Updated by Ward Vandewege over 4 years ago
- Subject changed from improve container logging when c-d-s has a problem with execution sbatch to improve container logging when c-d-s has a problem with sbatch execution
Updated by Ward Vandewege over 4 years ago
- Target version changed from To Be Groomed to 2020-05-20 Sprint
Updated by Tom Clegg over 4 years ago
- Related to Bug #13581: Report node not satisfiable users to the user added
Updated by Tom Clegg over 4 years ago
Shadowed variable bug.
16391-sbatch-error-logging @ 79d0b2480c207c12fd229403384881cd53a234e8 -- developer-run-tests: #1854
Updated by Lucas Di Pentima over 4 years ago
Isn't this update also shadowing the variable? Shouldn't be just "switch err.(type) {
"?
diff --git a/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go b/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
index 36bcef4f2..4115482d8 100644
--- a/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
+++ b/services/crunch-dispatch-slurm/crunch-dispatch-slurm.go
@@ -280,7 +280,8 @@ func (disp *Dispatcher) runContainer(_ *dispatch.Dispatcher, ctr arvados.Contain
cmd = append(cmd, disp.cluster.Containers.CrunchRunArgumentsList...)
if err := disp.submit(ctr, cmd); err != nil {
var text string
- if err, ok := err.(dispatchcloud.ConstraintsNotSatisfiableError); ok {
+ switch err := err.(type) {
+ case dispatchcloud.ConstraintsNotSatisfiableError:
var logBuf bytes.Buffer
fmt.Fprintf(&logBuf, "cannot run container %s: %s\n", ctr.UUID, err)
if len(err.AvailableTypes) == 0 {
@@ -296,7 +297,7 @@ func (disp *Dispatcher) runContainer(_ *dispatch.Dispatcher, ctr arvados.Contain
}
text = logBuf.String()
disp.UpdateState(ctr.UUID, dispatch.Cancelled)
- } else {
+ default:
text = fmt.Sprintf("Error submitting container %s to slurm: %s", ctr.UUID, err)
}
log.Print(text)
Updated by Tom Clegg over 4 years ago
Isn't this update also shadowing the variable? Shouldn't be just "
switch err.(type) {
"?
Yes and no: err is shadowed by a copy of the original err. switch err.(type)
wouldn't shadow err with the specific-type value, so err.AvailableTypes
wouldn't compile.
(In the previous code, the type assertion would return either err,true
or nil,false
, so the "else" case would always have err==nil instead of [a copy of] the original err.)
https://tour.golang.org/methods/16 https://play.golang.org/p/P88oNG4IP9u
Updated by Lucas Di Pentima over 4 years ago
Thanks for the example, this LGTM!
Updated by Anonymous over 4 years ago
- % Done changed from 0 to 100
- Status changed from New to Resolved
Applied in changeset arvados|0e802921cf0a4e6d1158df806d2a5dbeee074299.