Bug #4314

[Crunch] Figure out why this job was marked Failed unexpectedly

Added by Bryan Cosca about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/24/2014
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

examples: qr1hi-8i9sb-cha7spydhjauvvq qr1hi-8i9sb-w5hjmuq7vicng11 qr1hi-8i9sb-vt7mb676a4htd6k

They seemed to have this error in common: error: Unable to allocate resources: Requested nodes are busy. Ward said that there were two crunch dispatchers running and shutting one down seemed to fix it.

When the jobs end, they usually have a 403 Permission error and cannot output to keep.


Subtasks

Task #4373: Diagnose and fixResolvedPeter Amstutz

Task #4457: Review 4314-crunch-token-expireResolved

Task #4718: Review 4314-trigger-job-updateResolvedRadhika Chippada


Related issues

Related to Arvados - Bug #4310: [Crunch] crunch-dispatch --jobs locking is brokenResolved11/06/2014

Related to Arvados - Bug #4334: [Crunch] crunch-dispatch should not allocate Jobs to nodes in the idle* SLURM stateResolved10/28/2014

Associated revisions

Revision a0ff0429
Added by Peter Amstutz about 6 years ago

Merge branch '4314-crunch-token-expire' refs #4314 refs #4310

Revision 7e4a1953 (diff)
Added by Peter Amstutz almost 6 years ago

Touch the "crunch_refresh_trigger" file when the state changes. This notifies
all crunch-job instances to check the cancelled and state flags, so if a
running job changes state unexpectedly, it will be treated as a cancellation. refs #4314

Revision 4211e34c
Added by Peter Amstutz almost 6 years ago

Merge branch '4314-trigger-job-update' closes #4314

History

#1 Updated by Radhika Chippada about 6 years ago

  • Target version set to Bug Triage

#2 Updated by Tom Clegg about 6 years ago

  • Target version changed from Bug Triage to Arvados Future Sprints

#3 Updated by Tom Clegg about 6 years ago

  • Subject changed from Failed Jobs still continue running to [Crunch] Once a job has failed, crunch-dispatch should not run it
  • Story points set to 0.5

#4 Updated by Tom Clegg about 6 years ago

  • Target version changed from Arvados Future Sprints to 2014-11-19 sprint

#5 Updated by Tom Clegg about 6 years ago

Could be a duplicate of #4310

#6 Updated by Ward Vandewege about 6 years ago

  • Assigned To set to Peter Amstutz

#7 Updated by Peter Amstutz about 6 years ago

There's two different errors here.

#8 Updated by Tom Clegg about 6 years ago

The timing here makes the #4310 explanation less than 100% convincing. What would make crunch-dispatch take any interest in a job that has had state=='Running' for 37 minutes? (Process suspended???)

2014-10-24_19:59:31 qr1hi-8i9sb-vt7mb676a4htd6k 14114  start
...
2014-10-24_20:37:11 qr1hi-8i9sb-vt7mb676a4htd6k 14114  Job state unexpectedly changed to Failed

#9 Updated by Tom Clegg about 6 years ago

4314-crunch-token-expire looks good to merge, but it doesn't explain the failures reported here, does it?

#10 Updated by Peter Amstutz about 6 years ago

Some additional sluthing shows that qr1hi-8i9sb-vt7mb676a4htd6k changed from "Running" to "Failed" at 2014-10-24T19:59:32Z which suggests that it was a result of a race between crunch-dispatchers, but crunch-job didn't notice it until the task had completed 35 minutes later.

#11 Updated by Tom Clegg about 6 years ago

  • Subject changed from [Crunch] Once a job has failed, crunch-dispatch should not run it to [Crunch] Figure out why this job was marked Failed unexpectedly

#12 Updated by Tom Clegg about 6 years ago

  • Target version changed from 2014-11-19 sprint to 2014-12-10 sprint

#13 Updated by Peter Amstutz almost 6 years ago

Finally figured this one out. crunch-job only checks the job state if the file listed in "CRUNCH_REFRESH_TRIGGER" has been touched recently. It does this for cancellations, but not for other state changes, so even though the job was marked "failed" almost immediately due to the crunch-dispatcher race, it didn't notice until it completed on its own. 7e4a195 fixes that (unexpected state changes will be treated as cancellations).

("crunch_refresh_trigger" is an unfortunate backchannel method of communicating from API server to crunch-job, in the future when we need crunch-dispatch to run on a separate instance from crunch-dispatch this will need to use websockets.)

#14 Updated by Radhika Chippada almost 6 years ago

Review feedback:

Discussed the one-liner update with Peter for background info, and the update looks go to me.

And, all api server tests passed.

My only comment was, since the update was so close to the comment "# TODO: Remove the following case block when old ..." and we agreed that we will create a separate ticket to clean the old code.

LGTM.

#15 Updated by Peter Amstutz almost 6 years ago

  • Status changed from New to Resolved
  • % Done changed from 67 to 100

Applied in changeset arvados|commit:4211e34c99a068e8beb0baa6522c655c35b47b20.

Also available in: Atom PDF