Project

General

Profile

Actions
Copy link

Bug #4903

closed

[Crunch] ensure that JOB_WORK and TASK_WORK directories exist for Docker jobs

Added by Tim Pierce over 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Category:
Crunch
Target version:
-
Story points:
-

Description

Stack trace from job qr1hi-8i9sb-1fmcoid6lfess9g:

1/5/2015 5:40:24 PM    compute20    1    task-print    0    Traceback (most recent call last):
1/5/2015 5:40:24 PM    compute20    1    task-print    0    File "/tmp/crunch-src/crunch_scripts/bwa-index", line 11, in <module>
1/5/2015 5:40:24 PM    compute20    1    task-print    0    ref_dir = arvados.util.collection_extract(collection = this_job['script_parameters']['input'], path = 'reference', decompress = False)
1/5/2015 5:40:24 PM    compute20    1    task-print    0    File "/usr/local/lib/python2.7/dist-packages/arvados_python_client-0.1-py2.7.egg/arvados/util.py", line 201, in collection_extract
1/5/2015 5:40:24 PM    compute20    1    task-print    0    lockfile = open(path + '.lock', 'w')
1/5/2015 5:40:24 PM    compute20    1    task-print    0    IOError: [Errno 2] No such file or directory: '/tmp/crunch-job/work/reference.lock'

Possible diagnosis: Crunch jobs are run with JOB_WORK and TASK_WORK environment variables set to temporary working directories. These are made available to jobs as arvados.current_job().tmpdir and arvados.current_task().tmpdir respectively. However, when these jobs run in Docker, the JOB_WORK directory is not guaranteed to exist.

Actions
Copy link
 #1

Updated by Tim Pierce over 9 years ago

  • Subject changed from crunch job fails to create tmpdir to [Crunch] crunch job fails to create tmpdir
  • Description updated (diff)
  • Category set to Crunch
Actions
Copy link
 #2

Updated by Tim Pierce over 9 years ago

  • Subject changed from [Crunch] crunch job fails to create tmpdir to [Crunch] ensure that JOB_WORK and TASK_WORK directories exist for Docker jobs
  • Description updated (diff)
Actions
Copy link
 #3

Updated by Tom Clegg over 9 years ago

crunch-job creates JOB_WORK and TASK_WORK in the docker container (as ephemeral docker volumes). It's possible this job was run by a very old (2014-10-xx) version of crunch-job due to a deployment problem which has since been fixed.

Actions
Copy link
 #4

Updated by Brett Smith about 9 years ago

  • Status changed from New to Resolved
  • Target version deleted (Bug Triage)

Confirmed that the crunch-job fix is now effective. Tested in job qr1hi-4zz18-d2f9i56o1m9dkr7.

Actions
Copy link

Also available in: Atom PDF