Project

General

Profile

Bug #12355

Updated by Peter Amstutz over 6 years ago

It is taking hours to complete.    For some reason the "sudo" command to run crunch-job is taking 75 seconds to start. 

 <pre> 
 2017-09-27_18:41:15.94921 dispatch: sudo -E -u crunch LD_LIBRARY_PATH= PATH=/var/lib/gems/ruby/2.3.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/gems/bin:/usr/local 
 /go/bin PERLLIB=/usr/src/arvados/sdk/perl/lib PYTHONPATH= RUBYLIB=/var/lib/gems/gems/bundler-1.15.4/lib GEM_PATH= /usr/src/arvados/sdk/cli/bin/crunch-job --job-api-token 2e6f0zp3ralzjqscwu38czli 
 w6353bgziuat739i74sy12vaav --job mqkj9-8i9sb-0p5a2vn0evw4oa9 --git-dir /var/lib/arvados/internal.git --docker-bin docker --docker-run-args --net=host 
 2017-09-27_18:41:15.94968 dispatch: job mqkj9-8i9sb-0p5a2vn0evw4oa9 
 2017-09-27_18:41:15.94970 dispatch: child 28624 start 2017-09-27_18:41:15 
 2017-09-27_18:41:17.91463  
 2017-09-27_18:41:17.91467 2017-09-27 18:41:17 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:17.91467 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:23.72848  
 2017-09-27_18:41:23.72852 2017-09-27 18:41:23 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:23.72852 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:29.56375  
 2017-09-27_18:41:29.56379 2017-09-27 18:41:29 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:29.56379 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:35.34087  
 2017-09-27_18:41:35.34090 2017-09-27 18:41:35 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:35.34090 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:41.18088  
 2017-09-27_18:41:41.18092 2017-09-27 18:41:41 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:41.18092 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:46.98528  
 2017-09-27_18:41:46.98531 2017-09-27 18:41:46 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:46.98531 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:52.81456  
 2017-09-27_18:41:52.81458 2017-09-27 18:41:52 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:52.81459 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:41:58.58492  
 2017-09-27_18:41:58.58495 2017-09-27 18:41:58 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:41:58.58495 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:04.37249  
 2017-09-27_18:42:04.37252 2017-09-27 18:42:04 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:04.37252 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:10.17346  
 2017-09-27_18:42:10.17349 2017-09-27 18:42:10 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:10.17350 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:15.96954  
 2017-09-27_18:42:15.96957 2017-09-27 18:42:15 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:15.96958 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:21.76766  
 2017-09-27_18:42:21.76769 2017-09-27 18:42:21 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:21.76769 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:27.53407  
 2017-09-27_18:42:27.53409 2017-09-27 18:42:27 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:27.53410 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:33.32951  
 2017-09-27_18:42:33.32953 2017-09-27 18:42:33 +0000 -- pipeline_instance mqkj9-d1hrv-f2qy2hjqsxph10c 
 2017-09-27_18:42:33.32954 cwl-runner mqkj9-8i9sb-j6koxtmv6dtazgd {:failed=>0, :running=>1, :todo=>0, :done=>0} 
 2017-09-27_18:42:36.65370 mqkj9-8i9sb-0p5a2vn0evw4oa9 !    28626    Sanity check is `docker ps -q` 
 </pre> 

Back