Arvados

Note that the lock failure log message is coming from crunch-job, not from crunch-dispatch. crunch-dispatch sees the job in the queue, hands it to crunch-job. crunch-job reports back, "I couldn't work on this because I couldn't lock it." crunch-dispatch assumes that means that some other dispatcher got to the job first, so it leaves the record alone in the queue. It refreshes the queue, sees the job still there, tries again, and the loop repeats.

I think the fundamental question here is, how did the job get back on the queue after finishing on the 30th?

qr1hi-57u5n-m61q51j3hx5kgzv is the first log entry for the job, on 19 January. It shows that the job was originally created in this odd state: created on 19 January, finished on 30 December, already locked by qr1hi-tpzed-ohrz0hejlvl9g1n, with a log and output set. But it's in Queued state. Right now my top guess is that something went odd with job reuse.

twp: short answer: I ran that job almost certainly by copying out the job definition from a previous job of Bryan's, editing the fields by hand, and running arv-crunch-job
twp: very probably without editing out all of the fields and assuming (hah!) that crunch-job would DTRT with any that it didn't need

We should probably make the Job model/controller stricter somehow, although exactly how is still up for discussion.

I just canceled the job to get it out of the queue and relieve the immediate pain. If you want to see how it looked originally, refer to the log mentioned in my previous comment.

• Certain job attributes (output, log) should only be allowed to change (including from null to non-null) when the job is in Running state.
• Certain job attributes (finished_at) should only be changed while state is changing (from Running to Complete/Failed)