Bug #4970

[Crunch] Cannot create directory `/tmp/crunch-job/task/compute14.1.keep': File exists

Added by Bryan Cosca over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
-
Start date:
01/12/2015
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

qr1hi-8i9sb-wcsmjzy28gz9xf7

...not sure what this error is but I've gotten hit by it twice. even on a job that was supposed to re-run automatically.


Related issues

Is duplicate of Arvados - Bug #4967: [Crunch] Doesn't cope well with FUSE mounts left hanging around after killing tasks with SIGKILLResolved01/21/2015

Associated revisions

Revision ef969ca8
Added by Brett Smith over 5 years ago

Merge branch '4967-crunch-mount-cleanup-wip'

Closes #4967, #4970, #5039.

Revision 7c34347e (diff)
Added by Brett Smith over 5 years ago

4967: API server bundle uses bugfixed crunch-job.

Refs #4967, #4970.

History

#1 Updated by Bryan Cosca over 5 years ago

now im consistently getting hit by it: qr1hi-8i9sb-6p0w9zkhy2brrb9 qr1hi-8i9sb-fbwxxfu3mtdbgd8

#2 Updated by Tim Pierce over 5 years ago

  • Target version set to Bug Triage

#3 Updated by Brett Smith over 5 years ago

  • Subject changed from Cannot create directory `/tmp/crunch-job/task/compute14.1.keep': File exists to [Crunch] Cannot create directory `/tmp/crunch-job/task/compute14.1.keep': File exists
  • Category set to Crunch

This looks very related to #4967. In this case, qr1hi-8i9sb-da6mla8d48nd631 was canceled, and the underlying process was killed by signal 9.

The call in question uses mkdir -p. We could still get this error if the path existed and was not a directory. I don't have a good idea about how that might have happened, but it seems like crunch-job should clean its own environment as necessary either way.

#4 Updated by Tom Clegg over 5 years ago

This seems to be what mkdir says when you mkdir a stale fuse mount.

tom@alcan:~$ mkdir /tmp/foo
tom@alcan:~$ sshfs zoloft:/foo /tmp/foo
^C
tom@alcan:~$ sshfs zoloft:/foo /tmp/foo
fuse: bad mount point `/tmp/foo': Transport endpoint is not connected
tom@alcan:~$ mkdir -p /tmp/foo
mkdir: cannot create directory ‘/tmp/foo’: File exists

#5 Updated by Brett Smith over 5 years ago

  • Target version deleted (Bug Triage)

#6 Updated by Brett Smith over 5 years ago

  • Status changed from New to Resolved

Applied in changeset arvados|commit:ef969ca8dabe571a9866a7b3b7c39098785022fa.

Also available in: Atom PDF